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ABSTRACT 


Force Transformation requires a much greater emphasis on testing joint 
warfighting capabilities. A unique challenge in assessing the effectiveness and suitability 
of systems in the joint environment is the multitude of possible interactions and outcomes 
in a system-of-systems construct. Because of resource constraints and the complexity of 
conducting live, virtual, and constructive testing in a joint mission environment, the Joint 
Test and Evaluation Methodology (JTEM) program is interested in determining if 
analytical techniques, like Modeling and Simulation, can be applied to understand the 
relationship between system-of-systems performance and joint mission effectiveness. As 
a proof of concept, a Network Enabled Weapon (NEW) was chosen as a framework for 
this study. This thesis uses an agent-based distillation, which is a type of computer 
simulation, to model the critical factors of interest in a NEW engagement without 
explicitly modeling all of the physical details. Using cutting-edge experimental design 
techniques, the computer model was run many tens of thousands of times, with the results 
being analyzed to determine the critical parameters required for mission success. The 
analysis determined key interactions in NEW system performance and provides JTEM 
with a framework for efficiently conducting testing in a live environment. Specifically, 
the results indicate sensor range of a third-party ground controller, target speed, NEW 


impact radius, and weapon accuracy as the key factors affecting system performance. 
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THESIS DISCLAIMER 


The reader is cautioned that the computer model presented in this research may 
not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs are free of computational and 
logical errors, they cannot be considered validated. Any application of these programs 
without additional verification is at the risk of the user. 

Additionally, the intent of this effort was to gain insight into factors affecting 
overall system performance. The research was not conducted, nor was the model 
designed, to prove or demonstrate the capabilities of specific weapon systems used as the 
framework for characteristics of the simulated entities. As such, the reader should not 
use this research as a measure of assessing the effectiveness or suitability of weapon 


systems currently under development. 
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EXECUTIVE SUMMARY 


Joint operations have become the mainstay of warfighting. Force Transformation 
requires the Test and Evaluation (T&E) community to place a much greater emphasis on 
testing joint warfighting capabilities. A unique challenge in assessing the effectiveness 
and suitability of systems in the joint environment is the multitude of possible 
interactions and outcomes in a system-of-systems construct. New and developing 
acquisition programs rely on interfaces with existing or future systems, quite possibly 
from separate services, to achieve mission success. Because of resource constraints and 
the complexity of conducting live, virtual, and constructive testing in a joint mission 
environment, the Joint Test and Evaluation Methodology (JTEM) program is interested in 
determining if analytical techniques, like Modeling and Simulation (M&S), can be 
applied to understand the relationship between system-of-systems performance and joint 
mission effectiveness. As a proof of concept for investigating this possibility, Network 
Enabled Weapons (NEWs) was chosen as a framework for further study. The NEW 
concept centers on the ability to identify, engage, and attack moving targets, within 
moments of their identification, through the use of in-flight target updates (IFTUs) across 
a Weapon Control Network (WCN). 

As acquisition systems like NEW are required to conduct more testing in the 
context of a joint mission, it will be essential that these tests be as efficient and useful as 
possible. With the complexity of the joint test environment, M&S is one of the most 
effective tools to help understand the environment, design an efficient and useful test, and 
to help investigate robust possibilities in the use of forces to accomplish operational 
tasks. This thesis used an agent-based distillation, which is a type of computer 
simulation, to model the critical factors of interest in combat without explicitly modeling 
all of the physical detail. Agent-Based Modeling (ABM) refers to a type of simulation 
made up of agents (or entities) that behave autonomously. These agents possess simple 
internal rule-sets for decision making, movement, and action. When combined with other 


entities in the model and subjected to stochastic conditions, the agents interact in ways 


X1X 


that are often reflective of large-scale system behavior.! In close coordination with 
JTEM and program offices responsible for networked weapons, the author and the model 
developer worked through numerous iterations of programming and debugging over the 
course of six months in order to refine the model and improve its validity. 

As with many complex endeavors, military conflicts typify an environment of 
autonomous or semiautonomous agents, uncertainty in behavior and outcomes, a wide 
range of operational inputs, and complex interactions between entities.2_ The combination 
of ABM with Data Farming offers an exploratory, analytical approach to broadly 
consider uncertainties associated with elements of warfare that might otherwise be too 
costly or time intensive to study with other means. 

Data Farming involves the exploration of simulation models across a wide range 
of settings for the agents’ characteristics and running the model a statistically significant 
number of times.3 Rather than taking a “trial and error” approach to experimental design 
(be it live or in the M&S environment), researchers often use specialized techniques to 
organize the myriad of possible parameter settings. The overall objective of the design is 
to maximize the information gained from a limited number of experimental runs. In his 
Ph.D. Dissertation at the Naval Postgraduate School, LTC Thomas Cioppa, USA, 
provides an approach to experimentation geared toward addressing the issue of efficient 
design and analysis techniques for complex simulation models. His research indicates 
that the use of orthogonal, or nearly orthogonal, Latin hypercubes with excellent space- 
filling properties enable efficient exploration of simulation models.* Unlike traditional 
factorial designs, which test only a few factors at a minimum number of levels, a space- 


filling design explores a broad landscape of factor settings, as indicated in Figure ES-1. 





! Susan M. Sanchez and Thomas W. Lucas, “Exploring the World of Agent-Based Simulations: 
Simple Models, Complex Analyses,” Proceedings of the 2002 Winter Simulation Conference, 
December 2002, p. 1. 


2 Thomas M. Cioppa, Thomas W. Lucas, and Susan M. Sanchez, “Military Applications of Agent- 
Based Simulations,” Proceedings of the 2004 Winter Simulation Conference, December 2004, p. 1. 


3 Gary Horne and Ted Meyer, “Data Farming: Discovering Surprise,” Proceedings of the 2004 Winter 
Simulation Conference, December 2004, p. 2. 


4 Thomas M. Cioppa, LTC, USA, “Efficient Nearly Orthogonal and Space-Filling Experimental 
Designs for High-Dimensional Complex Models,” Ph.D. Dissertation, Naval Postgraduate School, 
Monterey, CA, September 2002, p. 9. 
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Figure ES-1. Scatterplot Matrix for a Selection of Eight Factors Exhibiting Excellent 
Space-Filling Properties of the Design. 


Figure ES-1 shows a two-dimensional projection for eight of the factor 
combinations in the single target (21-factor) design and exhibits outstanding space-filling 
properties. As a separate indicator of design efficacy, the orthogonality of the design 
improves the analysis of model results by decreasing correlation among regression 
coefficient estimates, and by allowing investigation of multiple high-order interactions, 


nonlinear relationships, and discontinuities in the response.> 


5 Thomas M. Cioppa, LTC, USA, “Efficient Nearly Orthogonal and Space-Filling Experimental 
Designs for High-Dimensional Complex Models,” Ph.D. Dissertation, Naval Postgraduate School, 
Monterey, CA, September 2002, p. 21. 
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For seven factors, this design approach allows the exploration of model effects 
across 17 uniformly distributed factor settings, in as few as 17 model runs. In 
comparison, a three-level, full factorial design of seven factors requires 37, or 
2,187 model runs, and would only examine three levels of each factor. When considering 
that this research examined the effects and interactions of more than 20 factors in each 
design, the significance is the difference between running 100 replications on a single 
laptop computer in five hours, or waiting until the sun burns out and never seeing 
the results. 

Using these cutting-edge, experimental design techniques, the computer model 
was run many tens of thousands of times, with each parameter having 257 settings varied 
uniformly over operationally viable ranges. The results were analyzed to determine the 
critical parameters required for mission success. In the case of a single moving target, 
indicative of a wheeled or tracked vehicle, the analysis indicates a significant 
time-distance interaction between the sensor range of the ground-based Joint Terminal 
Attack Controller (JTAC) and the speed of the target. Specifically, when the target speed 
is less than 13.2 meters/sec (approximately 30 mph) and the JTAC sensor range exceeds 
2,117 meters, the model indicates an 80% improvement in target kills, regardless of the 
other parameter settings. Moreover, when a combination of these two parameters is 
constrained across a realistic, but time-sensitive range, the model indicates that the 
amount of time taken by the decision authority to issue a Close Air Support (CAS) 
request and the speed at which the launch aircraft flies to engage the target provide the 
most improvement in mission success. 

This result is illustrated through use of a partitioning tree in Figure ES-2. The 
partitioning (or regression) tree is a statistical analysis approach aimed at identifying the 
most critical factors affecting the response variable—in this case, the proportion of target 
kills. Each block lists the factor name, the total number of data points contained in the 
branch, and the mean value and standard deviation of those points. To perform each 
“split” of the tree, the statistical software recursively divides the factor that most 
significantly separates the means by examining the differences in sums of squares. As 


shown in the figure, the ratio of JTAC sensor range to target speed is the most critical 


XXil 


factor in the model for the first two splits (albeit at different threshold values). If the ratio 
is between 197 and 305 seconds, then Command and Control decision time and aircraft 
speed account for the next significant splits, at the values shown. It should be noted that 
while the ratio variable of range to speed provides a meaningful way to conduct the 
present analysis, the variable itself is notional in the sense that it depends on the context 


in which the specific scenario was developed. 
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Std Dev 0.418791 StdDev —_0.2409168 
































Figure ES-2. Partition Tree for the Single Mobile Target Scenario with Number of Kills 
as the Response and a New Explanatory Factor Created from Dividing JTAC Sensor 
Range by the Target Speed. 


To test the system’s ability to engage a subsequently identified high-value target 
(HVT)—possibly with a redirect of a NEW already in flight—scenarios were run with 
two targets in the battlespace. The second target possesses the characteristics of a 
dismounted individual on patrol. The model indicates different results for the kill rate of 
the HVT, depending on whether or not the launch aircraft contains a load of two NEWs 
or just one. In the case of one weapon, for a JTAC sensor range of less than two 


kilometers, the kill rate of the HVT is shown to improve by nearly 82% if the IFTU 
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interval is less than 50 seconds. In the case where two weapons are available, the most 
important factors affecting the HVT kill rate are specific to the weapon itself. Namely, if 
the impact radius exceeds 5.4 meters and the probability of kill for a target within the 
blast radius is greater than 0.92, then the overall kill rate approaches 95%. 

While relatively simple in design, the simulation model provides a realistic 
depiction of operational scenarios and the interactions of systems within the NEW 
construct. Over the course of this research, the author consulted with JTEM personnel 
and subject matter experts within the NEW development community to discuss model 
functionality and the ranges of settings for model parameters. The author and the model 
co-developer worked through numerous iterations of programming and debugging over 
the course of six months in order to refine the model and improve its performance. 

The results of the analysis determined key interactions in NEW 
system-of-systems performance. Additionally, when considered in context of the 
scenarios developed, the model provides insight for program managers trying to 
understand the required performance characteristics of systems in development. Most 
importantly, the research indicates that ABM, especially when combined with efficient 
design principles, can yield a method to quickly analyze a complex system-of-systems 
construct and provide JTEM with a framework for effectively conducting testing in a 


live environment. 
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I. INTRODUCTION 


We must, therefore, be confident that the general measures we have 
adopted will produce the results we expect. 


—Karl von Clausewitz 


A. BACKGROUND 


Joint Vision 2020 was published by the Joint Chiefs of Staff to describe the 
operational concepts necessary to achieve success in our Nation’s future conflicts. An 
uncertain future creates a broad range of threat possibilities and requires an investment in 


technologies and new military capabilities.® 


The Secretary of Defense’s Quadrennial Defense Review (QDR), published 
30 September 2001, indicates that the transformation required to meet future threats is 
neither an end point itself, nor a single weapon system or strategy. Rather, it is the 
process used to achieve warfighting capabilities and is based on the following four 
Transformational Pillars: 

Strengthening joint operations through standing joint task force 


headquarters, improved joint command and control, joint training, and 
expanded joint force presence policy; 


Experimenting with new approaches to warfare, operational concepts and 
capabilities, and organizational constructs such as standing joint forces 
through wargaming, simulations and field exercises focused on emerging 
challenges and opportunities; 


Exploiting U.S. intelligence advantages through multiple intelligence 
collection assets, global surveillance and reconnaissance, and enhanced 
exploitation and dissemination; 


6 Joint Chiefs of Staff, Joint Vision 2020, United States Department of Defense, Washington, D.C., 
June 2000, p. 1. 


Developing transformational capabilities through increased and wide- 
ranging science and technology, selective increases in procurement, and 
innovations in DoD processes.’ 


In line with the pillar of experimentation, the QDR further states: “While 
transformation offers U.S. forces the promise of revolutionary capabilities, the products 
of this transformation must be tested thoroughly before they are deployed.’ In his 
National Military Strategy, the Chairman of the Joint Chiefs of Staff (CJCS) indicates 
that for the United States to maintain our advantage on the battlefield, we must transform 
our military by “combining technology, intellect and cultural changes across the joint 
community.”? The roadmap for the future of the U.S. military is clearly one requiring a 
transformation of capability and a need to test our ability to achieve success in a 


joint environment. 


Complexity arises because the systems participating in a joint mission will 
interact and evolve in ways that are not easily predictable. Systems are designed to meet 
specific requirements. Developmental testing and, to a degree, operational testing, are 
designed to evaluate whether the systems can meet those requirements. As those systems 
are given to operators for training and employment, they will find better ways to use the 
system’s inherent capabilities. Operators will also find ways to work around capability 
limitations. Tactical decision makers will employ those systems in combinations and in 
conditions not anticipated by the designers and testers—often in response to changes in 
tactics of an evolving, adaptive adversary. This is especially true when different systems, 


with different capabilities, are employed in a joint environment. 
B. TEST AND EVALUATION IN A JOINT ENVIRONMENT 


Joint operations have become the mainstay of warfighting. Force Transformation 


requires the Test and Evaluation (T&E) community to place a much greater emphasis on 


7 United States Department of Defense, Quadrennial Defense Review Report, Washington, D.C., 
30 September 2001, p. 32. 


8 Ibid., p. 41. 


9 Chairman of the Joint Chiefs of Staff, National Military Strategy of the United States of America, 
United States Department of Defense, Washington, D.C., 2004, p. 15. 
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testing joint warfighting capabilities developed in response to the Joint Capabilities 
Integration and Development System (JCIDS) process. T&E must ensure that our 
combatant commanders (CoComs) can rely on equipment (existing and future) to operate 
together effectively without introducing problems to the warfighters.!° 

Conducting live testing is further complicated by the limitations of defense 
budgets. Of the over $439 billion in the 2007 Department of Defense (DoD) budget, a 
significant portion, about $73 billion is devoted to Research, Development, Test, and 
Evaluation (RDT&E). Only $181 million, about one-quarter percent of the RDT&E 
amount, however, is devoted to Operational Test and Evaluation (OT&E). This amplifies 


the fact that operational testing must be structured as efficiently as possible.!! 


In 2003, the Defense Science Board Task Force Report on Enabling Joint Force 
Capabilities stated: 


. a network-centric approach based on a jointly developed network 
architecture remains essential if we are to field forces that can (1) respond 
quickly to a wide range of contingency demands and (2) act decisively 
from the outset against adaptive and resourceful adversaries. This requires 
full-capability, highly integrated joint land, sea, air, and space forces. !2 


Figure 1 depicts the flow of guidance leading to the need for integrated testing in 
the joint environment. The Transformation Planning Guidance, published by the United 


States Secretary of Defense (SecDef) in April 2003, states: 


As the Department transforms to a joint concept-centric approach for 
operational planning and capabilities development, we need integrated 
architectures that define the specific parameters of the requisite joint 
capabilities. A Joint Test and Evaluation Capability is needed to test the 


10 Director, Operational T&E, Testing in a Joint Environment Roadmap Strategic Planning Guidance, 
United States Department of Defense, Final Report, November 2004, p. vii. 


11 DoD budget data obtained from www.defenselink.mil/comptroller/defbudget/fy2007/index.html; 
site last accessed in May 2007. 


12 Defense Science Board, Report of the DSB Task Force on Enabling Joint Force Capabilities, 
August 2003, p. 1. 


capabilities in a realistic joint environment. Test and evaluation in a joint 
context will reveal whether or not the integrated architectures present a 
viable application of warfighting capabilities.!3 





Transformation 
Planning Guidance 


- Joint “concept-centric” 
approach for capability 
development 

- Integrated architectures 
define parameters of 
joint capabilities 

- Need to test capabilities 
and architectures in a 
realistic joint 
environment 


Signed April 2003 








Building the Need for T&E Transformation !4 


Figure 1. 


The Strategic Planning Guidance in March 2004 further delineated the need for a 
transformation in T&E and instructed the Director, Operational Test and Evaluation 
(DOT&E), to conduct a study to develop a roadmap on the way ahead for T&E 
transformation.!5 This roadmap was signed and published in November 2004, and it 
addressed three major issues: (1) The need to construct a joint environment suitable for 
testing applications from the mission requirements defined by the JCIDS capability 


planning process; (2) the requirement to use this environment across the entire acquisition 





Strategic Planning 
Guidance 


- Need realistic T&E ina 
joint operational context 

- Directed DOT&E to 
develop a roadmap to 


identify changes 
necessary to ensure 
T&E is conducted ina 
joint environment to 
enhance fielding of 
joint capabilities 


Signed March 2004 











Testing in a Joint 
Environment 
Roadmap 


- Build Joint Mission 
Environment from 
mission requirements 
defined by JCIDS 

- Required for entire 
acquisition process, not 
additional test 

- Defines infrastructure 
required: 

-- Network connectivity 

-- Service environments 

-- Program-specific 
Signed November 2004 


13 Secretary of Defense, Transformation Planning Guidance, United States Department of Defense, 


Washington, D.C., April 2003, p. 20. 


14 Bileen Bjorkman, Col, USAF, “JTEM Overview Briefing,” March 2006, slide 3. 
15 Strategic Planning Guidance for FYs 2006-2011, CLASSIFIED; not releasable to the public. 
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process, not as an additional test requirement; and (3) the required infrastructure, to 
include network connectivity, service-environments, and program-specific modeling 


and simulation. !© 
Cc. JOINT TEST AND EVALUATION METHODOLOGY TEST CASE 


Chartered by DOT&E with accomplishing the task of putting the roadmap into 
action, the Joint Test and Evaluation Methodology (JTEM) program was established in 
January 2006. Their overall goal is to develop a consistent approach to T&E in the joint 
environment. Among their main challenges are that processes and methods for designing 
and executing tests of system-of-systems are not well defined or understood, nor is there 
a clear understanding of how to assess system performance pertaining to capabilities 
supporting joint missions.!7 Due to resource constraints and the complexity of 
conducting live, virtual, and constructive testing in a joint mission environment, JTEM is 
interested in determining if analytical techniques, like modeling and simulation, can be 
applied to understand the relationships between system-of-systems performance and joint 
mission effectiveness. As a proof of concept for investigating this possibility, the 


Network Enabled Weapon (NEW) was chosen as a framework for further study. 


The NEW concept was born out of the requirement to create a network-centric 
integrated system. The scope of this research effort will be restricted to a test scenario 
for the current NEW concept of a sub 500-Ib class, nonpowered, guided bomb with data 
link capabilities and several guidance modes intended to attack moving targets in adverse 


weather and high-threat environments from the launch platform. 


As NEW systems are required to conduct more testing in the context of a joint 
mission, it will be essential that these tests be as efficient and useful as possible. With 


the complexity of the joint test environment, Modeling and Simulation (M&S) is one of 


16 Director, Operational T&E, Testing in a Joint Environment Roadmap Strategic Planning Guidance, 
United States Department of Defense, Final Report, November 2004, pp. 1-3. 


17 Eileen Bjorkman, Col, USAF, “JTEM Overview Briefing,” March 2006, slide 7. 
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the most effective tools to help understand the environment, design an efficient and 
useful test, and to help investigate robust possibilities in the use of forces to accomplish 


operational tasks. 


This thesis will use an agent-based distillation, which is a type of computer 
simulation that is used to model the critical factors of interest without explicitly modeling 
all of the physical details. Chapter II provides a description of the operational system-of- 
systems construct and a discussion on model development. In Chapter III, the 
experimental design methodology will be introduced, along with a description of 
scenarios based on the NEW construct. These scenarios will be replicated in the 
simulation tool and then analyzed. The analysis process will use a technique called data 
farming. This involves using high-speed computing to run the simulations thousands of 
times, while simultaneously varying many input parameters. Using cutting edge 
experimental designs developed at the Naval Postgraduate School, the data resulting from 
these simulations will be analyzed and presented in Chapter IV in order to identify 
critical factors, interactions, and thresholds. The results of the statistical analysis will 
then be extended to conclusions in Chapter V about the operational context and used to 
support the development of a joint test plan for NEW operations. It is envisioned the 
results of this research will directly support the planning and operation of JTEM test 


events for NEW operations—beginning as early as summer, 2007. 


I. SYSTEM DESCRIPTION AND MODEL DEVELOPMENT 


The general who wins a battle makes many calculations in his temple 
before the battle is fought. The general who loses a battle makes but few 
calculations beforehand. Thus do many calculations lead to victory, and 
few calculations to defeat: how much more no calculation at all! It is by 
attention to this point that I can foresee who is likely to win or lose. 


—Sun Tzu, The Art of War 


A. CHALLENGES FOR TESTING IN A JOINT MISSION ENVIRONMENT 


A unique challenge in assessing the effectiveness and suitability of systems in the 
joint environment is the multitude of possible interactions and outcomes in a 
system-of-systems construct. New and developing acquisition programs rely on 
interfaces with existing or future systems, quite possibly from separate services, to 


achieve mission success. 


Focus on Joint Mission Effectiveness (JMe) has increased emphasis on joint and 
coalition forces, integrated operations, network-enabled warfare, capabilities-based 
planning, etc. Test and Evaluation (T&E) to ensure that systems can operate in this new 
environment thus becomes more important, and increasingly complex. A corollary 
requirement is the analysis of JMe to assist in test planning, to understand and quantify 
the contribution of individual systems to the accomplishment of that mission, and assess 


the degree to which the mission can be accomplished. 


Within the Joint Mission Environment (JME), existing capability gaps impede 
successful execution of Joint Force Commander Joint Targeting Cycle processes against 
fixed, mobile, and relocatable planned and time-sensitive targets (TSTs). Flexible and 
responsive network operations across command, control, and communications systems, 
and intelligence, surveillance, and reconnaissance infrastructure are critical to 


successfully prosecuting these targets across multiple mission areas. 


The concept of Precision Engagement typifies this system-of-systems approach 
that “enables our forces to locate the objective or target, provide responsive command 
and control, generate the desired effect, assess our level of success, and retain the 


flexibility to reengage with precision when required.” !8 
B. NETWORK ENABLED WEAPONS (NEWs) 


Unlike some other precision-guided munitions (PGMs), the emphasis behind 
NEWS is the ability to attack stationary and mobile targets within moments of their 
detection. The capability to engage moving targets is gained through use of an onboard 


seeker and post-release communications network, enabling the following attributes: 


e In-flight target updates and retargeting provides target location update 

e Post-release communications acknowledgement provides positive 
weapon control 

e Weapon in-flight status provides feedback of weapon location and 
weapon status 

e Weapon abort provides a weapon disabling feature 

e Bomb impact assessment provides pre-impact target damage 
information!9 


NEWs achieves capability against mobile targets through updates from the launch 
aircraft or third-party targeting system via a weapon control network (WCN). 


The base network configuration for this research will include three active nodes— 
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the strike aircraft, the weapon, and the “weapon controller.” The weapon controller can 
refer to either the strike (i.e., launch) aircraft or a third-party Joint Terminal Attack 
Controller (JTAC). A multitude of possible mission scenarios exist, but an example of a 
particular construct is provided in Figure 2. The lightning bolts represent networked 
communications between the entities involved. Through communications with the 


Combined Air Operations Center (CAOC), an identified target gets assigned to a mission 





18 Joint Chiefs of Staff, Joint Vision 2010, United States Department of Defense, Washington, D.C., 
1997, p. 21. 


19 Headquarters, Air Combat Command, “Joint Enabling Concept for SDB II,” Draft, p. 7-1, FOUO; 
not releasable to the public. 


typical of the concept of operations for a NEW. During flight of the weapon, target 
location updates are passed to the guidance system of either the aircraft or the NEW in 
order to maneuver the munition to the target. This networked capability is what enables 
the engagement of moving targets without using existing technologies like laser-guided 


identification or camera-guided flight. 





CAOC 


~~ 


Target 











Figure 2. Guiding a NEW to the Target. 


Cc, AGENT-BASED MODELING (ABM) 


One approach towards gaining knowledge about the possible interactions 
associated with the NEW system-of-systems is through the application of ABM and 
Data Farming. ABM refers to a type of simulation made up of agents (or entities) that 
behave autonomously. These agents possess simple internal rule-sets for decision 


making, movement, and action. When combined with other entities in the model and 
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subjected to stochastic conditions, the agents interact in ways that are often reflective of 
large-scale system behavior.2° Data Farming involves the exploration of simulation 
models across a wide range of settings for the agents’ characteristics and running the 


model a statistically significant number of times.?! 


In their report to JTEM, researchers for Referentia Systems, Inc.?? described 


ABM and Data Farming: 


ABM can be viewed as a mathematical function; for a unique set of input 
(including any specific settings of the seeds for the random number 
generators used) there is only one output, both of which may be 
multidimensional. The goal of Data Farming is to understand that 
mapping, which may be very non-linear, to explore and identify the 
relationships between the input parameters and the output, to identify 
unexpected relationships, and to discover and characterize regions of the 
input space where discontinuities may exist.?3 


The overall objective for the use of Data Farming within JTEM is to extract as 
much information as possible from the models for each test scenario. It is envisioned that 
ABM will serve as a tool for refining the test space of possible parameters and parameter 


settings and narrow the focus for actual live test events.24 


There have been significant amounts of research regarding the applicability of 
ABM in the military context.2> As with many complex endeavors, military conflicts 
typify an environment of autonomous or semiautonomous agents, uncertainty in behavior 


and outcomes, a wide range of operational inputs, and complex interactions between 


20 Susan M. Sanchez and Thomas W. Lucas, “Exploring the World of Agent-Based Simulations: 
Simple Models, Complex Analyses,” Proceedings of the 2002 Winter Simulation Conference, December 
2002, p. 1. 


21 Gary Horne and Ted Meyer, “Data Farming: Discovering Surprise,” Proceedings of the 2004 
Winter Simulation Conference, December 2004, p. 2. 


22 Researchers and analysts, widely regarded as pioneers in the application of ABM combined with 
Data Farming. 


23 Gary Horne, Steve Upton, and Lawton Clites, Joint Test and Evaluation Data Farming Project: 
Initial Feasibility and Framing, Final Report under Contract NO0164-05-D-6656 TO 005, 31 January 2007, 
p. 15. 


24 Thid., p. 24. 
25 See Recent Theses and Papers at http://harvest.nps.edu; last accessed in April 2007. 
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entities.2° ABM offers an exploratory analytical approach to broadly consider 
uncertainties associated with elements of warfare that might otherwise be too costly or 


time intensive to study with other means. 
D. MODEL DEVELOPMENT 


Under a research contract sponsored by JTEM, Referentia Systems, Inc. 
developed an ABM for the system-of-systems concept. Built in the MASON 
programming language?’ and dubbed “TheTester,” the model contains the functionality 
desired for studying the complex interactions within a NEW scenario. In close 
coordination with JTEM and the program offices responsible for networked weapons, the 
author and the model developer worked through numerous iterations of programming and 
debugging over the course of six months in order to refine the model and improve its 
performance. Due to the speed of the model runs, and the graphical user interface 
implemented by the developer, the author was able to provide timely feedback and 
corrective suggestions to fix output anomalies. The types of agents contained in the final 


model used in this research effort include:28 


e Mobile and stationary targets of various classifications 

e Strike aircraft 

e NEWs 

e JTAC— ground-based and analogous Unmanned Aerial Vehicle (UAV) 
° WCN 

e Command and Control (C2) Center?? 


At the start of the simulation, stationary, mobile, or pop-up targets are placed on a 
three-dimensional grid through use of an input file. When a target enters the sensor range 


of a JTAC agent, a stochastically governed detection may occur. Upon detection, the 


26 Thomas Cioppa, Thomas W. Lucas, and Susan M. Sanchez, “Military Applications of Agent-Based 
Simulations,” Proceedings of the 2004 Winter Simulation Conference, December 2004, p. 1. 


27 See http://www.cs.gmu.edu/~eclab/projects/mason/; last accessed in April 2007. 
28 Each of these agents and their parameters are explained in further detail in Chapter III. 


29 Gary Horne, Steve Upton, and Lawton Clites, Joint Test and Evaluation Data Farming Project: 
Initial Feasibility and Framing, Final Report under Contract NO0164-05-D-6656 TO 005, 31 January 2007, 
p. 11. 
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JTAC relays a close air support (CAS) request through the C2 network to an available 
strike aircraft. The aircraft proceeds toward the target coordinates given by the JTAC 
until it reaches a predetermined launch (1.e., stand-off) distance. During flight, the strike 


aircraft receives target location updates directly from the JTAC. 


When the strike aircraft reaches the launch distance, the NEW is released if the 
target location information is not “stale.” If this condition is not satisfied, the strike 
aircraft either returns to base or remains on course toward the last known target position 


and, after a set interval, attempts another launch. 


Upon successful launch of the weapon, the NEW receives in-flight target updates 
(IFTU) of the target’s location until it gets within internal sensor range of the target. At 
that point, the weapon proceeds autonomously towards the target, in accordance with the 
rule set for its parameter settings. At any point during the pre- or post-launch sequence, 
if a target of higher “value” (HVT) is sensed by the JTAC, then the HVT becomes the 
target of primary interest. If post-launch, the JTAC will query the NEW through the 
WCN to determine if it has the capability to reach the HVT. If it does, then the NEW 
will redirect its course and attempt to destroy the HVT. Otherwise, the NEW proceeds 
toward the original target. In either case, once the NEW goes into autonomous seeker 
mode, the JTAC is no longer required to provide IFTUs, and therefore continues sensing 


for new targets—possibly including the original target in the case of a redirect. 


The main indicator of whether the engagement achieved the desired JMe centers 
on the concept of probability of a single-shot-kill. In other words, was the best available 
target destroyed in a timely manner with a single NEW? As will be explained in the next 
chapter, there are numerous dynamics affecting the outcome for each set of factor 
settings. The goal will be to determine which of those factors (and factor settings) most 
influences “success” of the mission in the model. From a thorough examination of the 
interactions within this joint system-of-systems model, it will be possible to draw 
recommendations regarding the best approach for live testing of a NEW mission— 


thereby achieving JTEM’s objectives for this research. 
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Chapter III provides the foundation for the experimental design, as well as a 
detailed description of the model agents, their individual parameters, and the ranges over 
which the parameters are varied. It concludes with a discussion of the model output 
(i.e., measures of effectiveness) and how that output will be used to address the key 


research objectives for this thesis. 
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Il. DESIGN OF EXPERIMENTS (DOE) 


The secret of all victory lies in the organization of the non-obvious. 


—Marcus Aurelius 

Marcus Aurelius, Emperor of the Roman Empire from 161-180 A.D., was 
considered one of the great Stoic philosophers.3° The Stoic mindset of “following where 
reason leads” typifies the leadership expressed in his quote above, for it is through an 
investigation of the potential nonobvious outcomes of decisions that allow informed 


leaders to improve their likelihood of success. 


Leaders in the Department of Defense (DoD) routinely make decisions involving 
expenditures of billions of dollars or involving life and death situations for personnel 
under their command. In an ideal setting, the outcomes of those decisions would be 
known with certainty. In reality, however, the vast number of factors with influence on 
the process or system under consideration often introduces a high degree of uncertainty in 
the outcome. To reduce the uncertainty, DoD authorities rely on experimentation to help 
understand the potential impacts of their decisions. Modeling and Simulation (M&S) is 
commonly used to replace live experimentation when real world systems are unavailable 


or, as is often the case, too expensive to devote to testing.>! 
A. GENERAL APPROACH 


Rather than taking a “trial and error” approach to experimental design (be it live 
or in the M&S environment), researchers often utilize specialized techniques to organize 
the myriad of possible parameter settings. The overall objective of the design is to 


maximize the information gained from a limited number of experimental runs. 





30 For more on Marcus Arelius and Stoic philosophy see http://en.wikipedia.org/wiki/Marcus_Aurelius 
and http://en.wikipedia.org/wiki/Stoicism, respectively. Last accessed in May 2007. 


31 Thomas W. Lucas and Susan M. Sanchez, “The Brave New World of Simulation Experiments for 
Defense and Homeland Security Applications,” Proceedings of the 2006 Joint Statistical Meetings, p. 1. 
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In his Ph.D. Dissertation at the Naval Postgraduate School, LTC Thomas Cioppa, 
USA, provides an approach to experimentation geared toward addressing the issue of 
efficient design and analysis techniques for complex simulation models. His research 
indicates that the use of orthogonal, or nearly orthogonal, Latin hypercubes with excellent 
space-filling properties enable efficient exploration of simulation models.32 Unlike 
traditional factorial designs, which test only a few factors at a minimum number of levels, 
a space-filling design explores a broad landscape of factor settings. The orthogonality of 
the design improves the analysis of model results by decreasing dependence among 
regression coefficient estimates, and by allowing investigation of multiple high-order 


interactions, nonlinear relationships, and discontinuities in the response. 


Figure 3 demonstrates the space-filling properties of the Nearly-Orthogonal Latin 
Hypercube (NOLH) design. The figure displays the pairwise combinations of factor 
settings in a two-dimensional projection. For example, the highlighted block shows the 
combination of factor settings this experiment would test for Factor B and Factor D. In 
three dimensions, the image would portray settings for three factors spaced efficiently 
throughout a cube. While we cannot visually comprehend the image in seven 
dimensions, the mathematics and the concepts still hold true. For seven factors, this 
design approach allows the exploration of model effects across 17 uniformly distributed 
factor settings, in a total of as few as 17 model runs. In comparison, a three-level, full 
factorial design of seven factors would require 37, or 2,187 model runs, and would only 
examine three levels of each factor. Considering that the intent is to replicate each design 
point hundreds or thousands of times to draw out the true nature of the stochastic 
response, the orders of magnitude difference in the number of model runs becomes 
increasingly significant. More poignantly, when considering that this research will 
examine the effects and interactions of more than 20 factors in each design, the 


significance is the difference between running 100 replications on a single laptop 


32 Thomas M. Cioppa, LTC, USA, “Efficient Nearly Orthogonal and Space-Filling Experimental 
Designs for High-Dimensional Complex Models,” Ph.D. Dissertation, Naval Postgraduate School, 
Monterey, CA, September 2002, p. 9. 


33 Ibid., p. 21. 
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computer in five hours, or waiting until the sun burns out (e.g., 2017 x 100 runs) and 
never seeing the results—and this is for a simulation model that takes less than one 


second to complete each run. 
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Figure 3. NOLH Scatterplot for 7 Factors and 17 Design Points Demonstrating 
Excellent Space-Filling Properties of the Design. 


B. MODEL PARAMETERS 


Each of the agents described in Chapter II possess one or more characteristics that 
define their behavior within the model. Depending on the specific scenario and the 
objectives for each design, some of the characteristics (synonymously referred to as 
parameters or factors) will be held to a constant value, while the majority are varied over 


a predefined range. Unless specifically stated otherwise, the factors are set to a particular 


iy 


value for each model run and do not fluctuate during that particular run. The following 
sections describe the parameters for each agent. A summary description is provided in 


Appendix A. 


1. Joint Terminal Attack Controller (JTAC) 


The JTAC is typically a ground-based controller deployed within the immediate 
vicinity of potential or known targets. He possesses sensor capabilities, communication 
devices, and equipment necessary to locate, identify, and track targets. Additionally, he 
is able to relay target location information to other agents on the Weapon Control 
Network (WCN). The agent could be used to replicate an air-based controller, such as an 
Unmanned Aerial Vehicle (UAV), by adjusting the parameters according to the 


characteristics of the entity being modeled. 


a. Speed 


If necessary, the JTAC can be assigned waypoints from his initial location 
to traverse during execution of the scenario. The speed setting controls how fast the 


JTAC moves between the assigned waypoints. Speed is measured in meters/second. 


b. Sensor Range 


The JTAC sensor range is a radial distance defining a circular 
representation of the agent’s ability to identify and track targets. The sensor 
characteristics are commonly referred to as “cookie cutter,’ indicating a defined 
probability of detection for targets within range and a probability of zero for targets 


outside the range. Sensor range is measured in meters. 


c. Probability of Detection (Pd) 


Targets within the JTAC’s sensor range are detected according to the 
assigned Pd. During each model time step, a random draw determines whether or not the 
target is detected. The logic within the model can be adjusted such that the Pd becomes 


one (certainty), after initial detection occurs, to simulate focused attention on the target. 
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d. Multi-Target Degradation Factor (MTDF) 


In scenarios containing more than one target, the JTAC’s ability to detect 
multiple targets may be degraded to represent the difficulty associated with simultaneous 


engagements. The MTDF is a factor between 0 and 1 applied multiplicatively to the Pd. 
e. Target Location Error (TLE) 


TLE defines the accuracy associated with the JTAC’s perception of the 
true location of the target. It is represented in the model by a Bivariate Normal 


distribution and is defined by two parameters, the mean and standard deviation.34 
f In-Flight Target Update (IFTU) Interval 


The JTAC does not continuously provide target location updates to other 
agents on the network. Instead, updates are transmitted at a frequency defined by the 


assigned /FTU—measured in seconds. 
2. Network Enabled Weapon (NEW) 


The NEW for this study is a nonpowered weapon launched from an aircraft. The 
combination of networked communications, control surfaces, and internal guidance 
systems give it the ability to redirect its path during flight and engage moving targets. 
Upon direction of the JTAC, the NEW is also capable of being reassigned to a 
higher-value target (HVT) during flight. Rather than modeling the complicated physics 
associated with determining a nonpowered weapon’s ability to reach a new target, a 
simplified approach was developed. Upon request from the JTAC for a redirect, the 
model calculates the three-dimensional slant distance to both targets and adjusts the 


weapon’s path only if the HVT is closer than the original target. In the case where the 


34 Jay L. Devore, Probability and Statistics for Engineering and the Sciences, 6th Edition, 2004, 
Thomson Brooks/Cole Publishers, 2004, p. 543. 
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weapon is not able to reach the HVT, a message is transmitted through the network to the 
JTAC indicating the redirect was denied, and the NEW continues flying toward the 


original target. 
a. Speed 


The true speed of flight is, again, a complicated physical process to model. 
The simplifying approach for this study is to vary the speed as a parameter across a 
typical range between maximum glide performance and minimum time to target. Every 
effort was made to consult with Subject Matter Experts (SMEs) in order to accurately 


represent weapon performance.35 Speed is measured in meters/second. 
b. Impact Radius 


The impact radius defines a circular area of lethality around the center of 
the bomb’s impact location. Upon impact, the model determines whether the target’s true 
location is within the area and, in combination with the probability of kill (Pk), assigns a 


target kill or miss accordingly. The impact radius is measured in meters. 
c. Sensor Range 


The sensor range identifies the point at which the NEW acquires 
autonomous recognition of the target. The performance within the model is similar to the 
“cookie cutter” approach described for the JTAC agent. Once the NEW is within this 
three-dimensional range to the target, it no longer receives IFTUs through the network; 


instead relying on its own sensor characteristics to guide it to the target. 
d. Target Location Error 


The 7LE is identical to the process described for the JTAC. The Gaussian 
mean and standard deviation parameters, however, will take on a different range of 


values in order to more closely represent NEW sensor capabilities. 


35 The author and model developer consulted in person with SMEs from Joint Command and Control 
for NEW and SDB II program offices, Eglin AFB, on 26 April 2007. 
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e. Probability of Detection (Pd) 


Once in autonomous seeker mode, the NEW relies on its own Pd to 
determine if it senses the target during a particular time step. Unlike the JTAC, however, 


the Pd is always a random draw, according to the parameter setting. 
f Probability of Kill (Pk) 


Combined with the impact radius, the NEW’s Pk value determines 
whether or not a target is killed. The Pk is limited to very high (1.e., favorable) values to 


replicate the likelihood of correct weapon detonation upon impact. 
3; Target 


Targets for this study are notional in that they do not have any distinguishing 
characteristics to differentiate them among typical NEW target sets. Targets can, 
however, be assigned a priority value in order to account for the presence of an HVT. 
Also, the targets may be assigned different Pd and Pk values for the JTAC and the NEW. 
The two parameters explicitly varied as part of the DoE are the target speed and the 


pop-up time. 
a. Speed 


For each scenario, the target(s) are assigned predetermined waypoints to 
traverse. The target may be set to cycle through the waypoints continuously or stop when 
it reaches the last one. The target speed is used to differentiate between categories of 
targets. For example, vehicular targets typically travel at much greater speeds than 


dismounted (i.e., foot patrol) targets. Speed is measured in meters/second. 
b. Pop-Up Time 


Each target in the scenario may be designated a particular time step in 
which it first appears. The pop-up time presents the opportunity to investigate system 


effectiveness against a HVT that may result in a redirecting a NEW already in flight. 
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4. Launch Aircraft 


The aircraft’s initial location can be varied within the model, but for the purposes 
of this study, the starting position is simulated as a Combat Air Patrol (CAP) orbit 
approximately 100 kilometers (km) south of the JTAC at an altitude of six km. After 
target identification, the aircraft agent receives an assigned mission from the 
Command and Control (C2) Center and is loaded with a predetermined number of NEWs. 
The aircraft is then directed toward the target by location updates relayed directly from 
the JTAC. In accordance with the parameter settings for a particular model run, the 
aircraft attempts to launch the NEW and then proceeds back to its starting position. The 
aircraft may, if equipped with remaining weapons, be directed at any point to conduct 


another mission. 


a. Speed 


The aircraft speed setting is varied over a range representative of a typical 
strike aircraft and is constant throughout a particular model run. Aircraft speed is 


measured in meters/second. 


b. Launch Distance, Update Requirement, Number of Attempts, 
and Reattempt Interval 


This section describes four different factors for the aircraft agent. The 
launch distance represents the stand-off distance where the aircraft will initially attempt 
to release the NEW. If the target location information has not been successfully relayed 
to the aircraft within the time period set by the update requirement factor, the aircraft will 
not launch the weapon. This is designed to limit the launch of a weapon when the target 
is no longer within sensor range of the JTAC. Based on the number of attempts setting, 
the aircraft may make subsequent attempts to launch the NEW, at a frequency governed 
by the reattempt interval. While awaiting multiple attempts to launch the weapon, the 


aircraft continues to fly toward the last known target location. 


me 


5. Weapon Control Network (WCN) 


The WCN is the host for the flow of communication between the NEW and JTAC 
agents. It is the heart of the capability to operate in a network-centric environment. 
While there are a multitude of potential issues and limitations affecting the transmission 
capability and interoperability of systems across any network, the model used for this 
research considers only two of the important factors. The range of the network is 


simulated to encompass the entire range of the battlespace. 
a. Reliability 


The reliability of the network is a parameter that governs the random 
probability of a successful message transmission. For the purposes of this research, 
messages transmitted are either fully received or not at all—there are no instances of 


partial (or garbled) communications. 
b. Latency 


Latency is a measure of how long it takes for a message to be transmitted 


across the network and is measured in seconds. 
6. Command and Control (C2) Center 


When the JTAC senses a target, the information is transmitted to the C2 Center in 
the form of a Close Air Support (CAS) request. The intent is to simulate a command 
element within the scenario with the authority to assign an aircraft equipped with a NEW 


to prosecute the target. 
a. Decision Time 


The only parameter currently in use for the C2 agent is the latency 
associated with the amount of decision time it takes to assign the mission to an available 
aircraft. The time is measured in seconds and effectively introduces a delay in between 


target identification and CAS mission authorization. 
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C. SCENARIO DEVELOPMENT 


99 


NEW missions typically fall into the category of “reactionary strike,” meaning 
targets of opportunity that involve little or no prior knowledge of the target’s exact 


location and/or route. The missions may involve moving or stationary targets. 


This research considers a scenario involving ground-based mobile targets and a 
JTAC as the post-launch weapon controller. One of the targets is envisioned as a 
wheeled vehicle that follows a prescribed route within the vicinity of the stationary 
JTAC. Depending on the settings for a particular model run, a second target of higher 


priority, simulating a dismounted combatant may “pop up” in the battlespace. 


As depicted in Figure 4, the JTAC is located at the center of the simulation 
battlespace. The vehicular target begins its route at a position 2,000 meters east and 
1,000 meters north of the JTAC. The target then follows a series of set waypoints at its 
designated speed. At some point in the scenario, the target will enter (and exit if not 
destroyed) the sensor range of the controller, initiating a CAS request once detected. In 
some scenarios, an HVT may appear to the southeast of the controller’s position. This 
target will cycle through a square pattern at a speed indicative of a human on foot patrol. 
Upon detection of the HVT, the JTAC will submit a new CAS request, which may result 
in either an attempt to redirect an inbound NEW or the launch of a second weapon, 


if available. 
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Target detection and Vehicle Target 
tracking based on -Start (2,000, 1,000) 
JTAC sensor range <Sa2/ -WP! (-300, 0) 
-WP2 (500, -1,000) 
-WP3 (-4,000, -7,000) 
HVT on foot patrol within 
~ 500 m of JTAC 
Vehicle Target 
Departs if not Killed 
F-18 flying CAP ~100 km 
a south of JTAC, 20k ft alt 
*not to scale 
Figure 4. Visual Depiction of the Battlespace Indicating JTAC Positioning 
and Target Routing. 


This scenario is based, in part, on mission descriptions found in the draft Joint 
Enabling Concept document for SDB II.3° The intent is to simulate a realistic operational 
setting that would offer the opportunity to test the NEW system-of-systems’ effectiveness 
against mobile targets and time-sensitive HVTs. Several effectiveness measures were 


developed in order to facilitate the assessment. 
D. MEASURES OF EFFECTIVENESS (MOEs) 


Because of resource constraints and the complexity of conducting testing in a 
joint mission environment, JTEM is interested in determining if analytical techniques, 
like modeling and simulation, can be applied to understand the relationship between 
system-of-systems performance and Joint Mission Effectiveness (JMe). Specifically, this 


research was designed to address the following questions: 


36 Headquarters, Air Combat Command, “Joint Enabling Concept for SDB II,” Draft, Section 8, 
FOUO; not releasable to the public. 
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e For a particular scenario, what are the potential performance parameters 
(and their applicable ranges) requiring testing to determine 
system capability? 


° Given that the ability to conduct live testing is resource constrained, what 
is the critical collection of parameter inputs to test? 


e For a particular construct, are there possible mixtures of operational forces 
or Tactics, Techniques, and Procedures (TTPs) that could have a 
significant impact on JMe? 


e Can Agent-Based Modeling (ABM) assist JTEM in determining 
approaches to Joint Test and Evaluation? 


The main measure used in this study to address the research questions is the 
number of target kills. The output files generated by the model provide a binary 
description of whether a particular model run resulted in a kill for each target type, as 


applicable. The pertinent output data generated by the model is presented in Table 1. 





























MOE Description 
Target Kill Binary kill indicator for each target type in each model run 
CAS Requests Total number of CAS requests in each model run 
NEW Launches Total number of NEW launches in each model run 
Abort — No Target Update [Mission abort — Launch aircraft did not have recent target info 
Redirected Redirect of the NEW occurred during mission 
Redirect Denied Redirect of the NEW denied during the mission 
Hit — No Kill NEW hit within impact radius of target, but no kill 
Outside Impact NEW missed the target 








Table 1. | Model Output Used as MOEs. 


E. EXPERIMENTAL DESIGNS 


The designs used for the research are based on the orthogonal and space-filling 
concepts previously discussed. Two NOLH design matrices>” were constructed—one for 
a scenario involving the vehicular target and one containing parameters for both the 


vehicular and pop-up HVT. Table 2 shows the parameter settings for the two target 


37 The spreadsheet tool used to build the designs can be found under the Software Downloads link at 
http://harvest.nps.edu.; last accessed in April 2007. 
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scenario. A total of 27 factors were varied over the ranges shown. Several factors, such 
as the pop-up time for the Mobile-1 target, are included in the table but were assigned a 
specific value consistent across all model runs. The one target scenario factor ranges are 
identical to those shown in Table 2, with the exception of the six highlighted parameters 


unique to the HVT scenario, which were excluded. 
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WCN_Rel Weapon Control Network Reliabilit 0.95 | 10 | n/: 
WCN_Lat Weapon Control Network Latenc | 3. ‘| seconds 


C2_DecTime Command and Control Decision Time 10 seconds 
Po Indicates factors not used in the one target scenario 








Table 2. Factor Settings and Ranges for the Two-Target Scenario. 


Figure 5 shows a two-dimensional projection for eight of the factor combinations 
in the single target (21-factor) design. As previously shown in the example at Figure 3, 


the design exhibits outstanding space-filling properties. 
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Figure 5. Scatterplot Matrix for a Selection of Eight Factors Exhibiting Excellent 
Space-Filling Properties of the Design. 


Table 3 shows the shows the correlation matrix for the same eight factors. As 
indicated in Cioppa’s research, the columns of the design matrix are “nearly orthogonal,” 
exhibited by the minimal correlation, in this case much less than +/— 0.03.38 The decision 
to display a selection of only 8 of the 21 factors included in the design was one of 
presentation and space. The concept and the design results extend similarly to an 


analysis of the complete experimental design matrix. 


38 Thomas M. Cioppa, LTC, USA, “Efficient Nearly Orthogonal and Space-Filling Experimental 
Designs for High-Dimensional Complex Models,” Ph.D. Dissertation, Naval Postgraduate School, 
Monterey, CA, September 2002, p. 23. 
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Correlations 


TgtC_Spd NEW_Spd NEW_ImpRad NEW_SnsrRng NEW_TLEmu AC_Spd AC_LnchDist C2_DecTime 





TgtC_Spd 1.0000 -0.0004 0.0003 0.0000 0.0000 -0.0002 0.0019 -0.0001 
NEW_Spd -0.0004 1.0000 0.0018 0.0017 0.0018 -0.0001 0.0005 -0.0006 
NEW_ImpRad 0.0003 0.0018 1.0000 -0.0002 0.0005 0.0002 0.0005 -0.0003 
NEW_SnsrRng 0.0000 0.0017 -0.0002 1.0000 0.0000 0.0013 -0.0003 0.0005 
NEW_TLEmu 0.0000 0.0018 0.0005 0.0000 1.0000 0.0009 0.0011 0.0005 
AC_Spd -0.0002 -0.0001 0.0002 0.0013 0.0009 1.0000 -0.0011 -0.0009 
AC_LnchDist 0.0019 0.0005 0.0005 -0.0003 0.0011 -0.0011 1.0000 -0.0010 
C2_DecTime -0.0001 -0.0006 -0.0003 0.0005 0.0005 -0.0009 -0.0010 1.0000 


Table 3. Pairwise Correlation Matrix Indicating Minimal Correlation Among 
the Columns of the Design. 


The NOLH approach used in this research created 257 unique design points, or 
excursions. Each excursion was replicated 100 times, for a total of 25,700 simulated tests 
for each scenario. Chapter IV follows with a discussion of the results and an explanation 


of the analytical tools used to process the simulation output data. 
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IV. RESULTS —- DATA ANALYSIS 


The purpose of computing is insight, not numbers. 


—Richard Hamming 


This chapter covers the analysis conducted to address the research questions. It is 
important to note that the intent of the design and analysis approach in this thesis was to 
stress the system-of-systems concept in ways that may not have been intended by the 
architects of specific entities within the construct. Specifically, while every attempt was 
made by the author to represent realistic characteristics of a Network Enabled Weapon 
(NEW), the overall objective was not to prove or demonstrate a certain level of 
capability. Rather, the intent was to gain insight into system performance from the 
standpoint of which factors (and at what settings) most significantly affects overall 
system behavior. Review of the results and analysis from this research must be kept in 
context with the description of the model and the scenarios, as previously discussed. 
Extrapolation of the results beyond the particular parameter settings used in the designs is 


discouraged—investigation into that realm remains a topic for future study. 
A. STATISTICAL SETUP 


Analysis of the model output was conducted with JMP Statistical Discovery 
Software, Version 5.1. JMP is a division of SAS, a leader in business intelligence and 
analysis, and is an accepted standard worldwide in the research, educational, and business 
fields.39 This section describes the procedures used to extract the model output into 
useful files for analysis. Also covered in this section is a general introduction to the 


statistical methods used to interpret the results. 


39 See http://www.jmpin.com.; last accessed in May 2007. 
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if Procedures for Extracting and Summarizing Model Output 


Each run of the model generated an output data file that contained the results of 
the Measures of Effectiveness (MOEs) discussed in Chapter III. In order to equate the 
output to the corresponding input variables for a specific design point, a Ruby? script 
was written that matched the design matrix to the output and created a single comma 
separated value file.4! Each row of the output.csv file contains an organized sequence of 
values for the input parameters, followed by the specific MOE values generated by the 
model as the response. In the case of an experiment consisting of 257 design points 


(DPs) and 100 replications, the output.csv file would contain 25,700 rows. 


Once the output was organized into an acceptable format, the data was imported 
into JMP for further analysis. The first technique involved combining the data into 
summary statistics. Since the same inputs were used in each replication of a specific DP, 
the use of the mean value of the MOE across all replications (in this case, 100) serves as 
a “sufficient statistic” and reduces the variance within a set of inputs while preserving all 


of the information regarding the true nature of the response.42 
2. Brief Explanation of Stepwise Regression and Partitioning Trees 


The two main tools used in this research to analyze the results are 
Stepwise Regression and Partitioning Trees. The theory behind each method is addressed 
here briefly, while the specific application of the approach is covered in more detail in the 


following sections, as results are presented. 


Regression analysis deals with the stochastic relationship between predictor 
variable(s) (also referred to as explanatory or independent variables), and at least one 
response variable of interest. The procedure is useful for making inferences about the 


effects changes to the inputs have on the output and provide a quantitative measure of the 





40 See http://ruby-lang.org/en.; last accessed in May 2007. 
41 Files listed with a .csv extension refer to comma separated value format. 


42 VK. Rohatgi, An Introduction to Probability Theory and Mathematical Statistics, John Wiley and 
Sons, Inc., June 1976, pp. 339-340. 
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extent of those effects.43 In JMP, Stepwise Regression is a selection within the model- 
fitting platform and is an approach to selecting a subset of effects for a regression model. 
It is often used when there is little theory to guide the selection of terms for a model, but 
the modeler believes the approach will provide a good fit. The approach has been used 
practically, however, for over 30 years to help reduce the size of models and predict 


many kinds of responses, including nonlinear interactions.“4 


Partitioning Trees provide a method to separate the predictor variables into sets of 
data that explain significant differences in the response. To perform each “split” of the 
tree, the statistical software recursively divides the factor which most significantly 
separates the means (in the case of continuous variables) by examining the differences in 
sums of squares. The method is often very useful, especially with large data sets, because 
there are no prior assumptions required regarding the underlying distribution of the 


response and, as will be shown, analysis of the results is very straightforward.45 
B. RESULTS FOR THE ONE TARGET SCENARIO 


The first experiment involved the examination of NEW system effectiveness for 
the case of one mobile target, a ground-based weapon controller, and an aircraft equipped 
with one NEW. The Design of Experiments (DoE) included 21 factors, 257 DPs, and 
100 replications of each DP. The complete DoE is provided in Appendix B. 


Figure 6 provides the initial split of a partition tree with proportion of target kills 
as the MOE. The top block indicates an overall 0.48 kill rate for all 257 DPs.46 
Increases in the proportion of target kills are considered an improvement in the 
response—shown as branching to the right for this particular tree. As the figure 
indicates, the most critical factor in the model for determining kill rate is the target’s 


speed. For the 147 DPs with target speed set less than 13.19 m/s (approximately 29.5 


43 Jay L. Devore, Probability and Statistics for Engineering and the Sciences, 6th Edition, 2004, 
Thomson Brooks/Cole Publishers, 2004, pp. 497-498. 


44 JMP User’s Guide, Companion Software to JMP 5.1, 2003. 
45 Tbid. 
46 The number of data points in each block will be shown by “Count” on the partition tree. 
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mph), the model indicates an improvement in target kills to 74%, opposed to 
approximately 14% for the other 110 cases. Successive splits of the partitioning tree 


occur sequentially for the parameter explaining the most variation in the response. 


Target speeds less than 
All Rows 13.19 m/s (approximately 
Count 57 29.5 mph) result in 
Mean 0.4830739 significant improvement in 
StdDev —_(0.4579518 system performance. 








Tgt_C_Spd>=13.19 Tgt_C_Spd<13.19 


Count 110 Count 147 
Mean 0.1390909 Mean 0.7404762 
Std Dev 0.3276117 Std Dev 0.3629603 








Figure 6. First Split of a Partition Tree for Target Kills as the MOE. 


The results shown in Figure 7 provide a more complete breakout of the factors 
impacting target kills. As previously mentioned, interpretation of the splits for this MOE 
is that branching to the right indicates an improvement in the response. Each box should 
also be viewed as conditional upon the parameter value(s) in the preceding levels of the 
tree. For example, the block in the bottom right corner of the tree contains 98 DPs where 
the JTAC sensor range is greater than or equal to 2117 meters and greater than or equal to 


1150 meters (which is trivial in this case), and where target speed is less than 13.19 m/s. 
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All Rows 
Count 257 
Mean 0.4830739 
Std Dev 0.4579518 








































































































































































































Tgt_C_Spd>=13.19 Tgt_C_Spd<13.19 
Count 110 Count 147 
Mean 0.1390909 Mean 0.7404762 
Std Dev 0.3276117 Std Dev 0.3629603 
NEW_SnsrRng<2469 NEW_SnsrRng>=2469 JTAC_SnsrRng<1150 JTAC_SnsrRng>=1150 
Count 80 Count 30 Count 20 Count 127 
Mean 0.056625 Mean 0.359 Mean 0.096 Mean 0.8419685 
Std Dev 0.219804 Std Dev 0.4505655 Std Dev 0.2843164 Std Dev 0.2532802 
JTAC_SnsrRng<2117 JTAC_SnsrRng>=2117 
Count 29 Count 98 
Mean 0.6506897 Mean 0.8985714 
Std Dev 0.4264382 Std Dev 0.1290808 
Figure 7. First Five Splits of the Partition Tree for Target Kills as the MOE.47 


Two important points are revealed in Figure 7. First, if the operational context 
includes targets traveling greater than the threshold speed of 29.5 mph, then the model 
indicates improvements in NEW system performance may still be achieved if the 
weapon’s internal sensor range is greater than approximately 2,500 meters. Secondly, for 
targets with speeds less than 29.5 mph, an increase in the JTAC’s sensor range beyond 
approximately 2.1 kilometers may result in an overall 80% improvement in target kills— 
shown by comparing an MOE value of ~0.90 in the lower right branch to the 0.48 value 
at the root node. The apparent significant impact of the time/distance relationships 


exhibited by the results warrants further exploration. 


Regression analysis was used to further investigate the parameters affecting the 
simulation model’s response of target kills. The first approach considered the main 
effects only—meaning only the factors themselves, without consideration of second order 
interactions and quadratics, were included in the analysis. Figure 8 shows the 
Stepwise Regression control panel within JMP. The figure is used to illustrate the 
settings chosen for the regression model. The user selects a probability to enter the 
model and a probability to leave. These levels indicate the statistical significance a 


potential regressor term must achieve for it to be considered in the model in a forward or 


47 Successive splits on the same parameter is often an indicator of a nonlinear effect. 
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backward step, respectively. The direction can either be specified as forward, backward, 
or mixed. The mixed setting was used to cause the process to alternate between forward 


(adding terms) and backward (removing terms) steps. 


Response: Mean(C-Kills) 
v Stepwise Regression Control 


Prob to Enter | 0.050) 
Prob to Leave Hee "Enter All _ 


Direction: Mixed v 


( Remove “Remove All) 
—Géo— Stop | Step — ( Make Model ) 





Figure 8. Stepwise Regression Control Panel in JMP. 


The summary of fit shown in Figure 9 illustrates the R-squared (R’) and adjusted 
R° values for the resulting model. These are statistical measures that indicate the 
proportion of variation in the response attributable to terms in the model, as opposed to 


random error. An R’ value of 1.0 would indicate the model perfectly fit the data. 


Summary of Fit 


RSquare 0.582328 
RSquare Adj 0.574007 


Root Mean Square Error 0.298896 
Mean of Response 0.483074 
Observations (or Sum Wats) 257 





Figure 9. Summary of Fit Resulting from Stepwise Regression for Main 
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The adjusted R’ value is especially useful in stepwise procedures because it 
provides a better comparison of “fit” across models by using the degrees of freedom in its 
computation.48 The adjusted R* value shown in Figure 9 of 0.57 will be considered a 


baseline against which to judge other regression models for this scenario. 


After conducting the stepwise reduction procedure, the terms with a statistically 
significant impact on the regression model are shown in Figure 10. The results are 
consistent with the findings for the partition tree—namely target speed and JTAC sensor 
range contributing the most to the response. The column labeled “Scaled Estimate” 
provides the scaled regression coefficients that could be used to calculate a value for the 
expected proportion of target kills for a given set of input variables. The scaling provides 
a meaningful way to compare coefficients whose corresponding factors may have 
different units. A clarifying example is provided in Appendix C. Calculations performed 
in this vein are only valid for parameter values within the range of factor settings used to 
generate the original set of data. The fact that the coefficient for target speed has a 
negative value indicates that increases in speed reduce the calculated value of the 


response, which is in line with intuition and previous findings. 


Scaled Estimates 


Continuous factors centered by mean, scaled by range/2 


Term Scaled Estimate Std Error t Ratio Prob>litl 
Intercept 0.4830739 0.018645 25.91 <.0001 
Tgt_C_Spd -0.497853 0.032169 -15.48 <.0001 
NEW_ImpRad 0.0654849 0.032162 2.04 0.0428 
NEW_PdC 0.0697697 0.032091 2.17 0.0306 
AC_Spd 0.0808914 0.032169 2.51 0.0125 
JTAC_SnsrRng 0.3137775 0.032169 9.75 <.0001 








Figure 10. Scaled Estimates of the Regression Coefficients in the Main Effects Only 
Stepwise Reduced Statistical Model for Target Kills as the MOE. 


48 Jay L. Devore, Probability and Statistics for Engineering and the Sciences, 6th Edition, 2004, 
Thomson Brooks/Cole Publishers, 2004, p. 580. 
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Running the stepwise reduction on all two-way interactions and 2nd order 
polynomial effects provides the next regression model. Figure 11 displays the scaled 
estimates of the coefficients. For this figure, the terms have been sorted based on the 
value of the scaled estimate. This selection of terms produced an adjusted R’ value 
of 0.63, which indicates a modest improvement in the explanatory power of the statistical 
model, compared to the model with only main effects considered. This model does, 


however, provide insight into the interactions between the parameters. 















































Scaled Estimates 

Continuous factors centered by mean, scaled by range/2 

Term Scaled Estimate Std Error t Ratio Prob>itl 
Intercept 0.4665227 0.039241 11.89 <.0001 
JTAC_SnsrRng 0.3138202 0.029916 10.49 <.0001 
(JTAC _TLEmu-55.0039)*(JTAC _TLEmu-55.0039) 0.1676852 0.05929 2.83 0.0051 
(AC_LnchIntrvl-20.0016)*(AC_LnchIntrvl-20.0016) 0.1306483 0.060957 2.14 0.0331 
(Tgt_C_Spd-12.0012)*(NEW_SnsrRng-2000.03) 0.1281817 0.055357 2.32 0.0214 
(NEW_PdC-0.92502)*(C2_DecTime-105.001) 0.1059225 0.054574 1.94 0.0534 
AC_Spd 0.0803891 0.029918 2.69 0.0077 
NEW_PdC 0.0716894 0.02985 2.40 0.0171 
NEW_ImpRad 0.0634571 0.02993 2.12 0.0350 
NEW_SnsrRng 0.0574092 0.029921 1.92 0.0562 
AC_LnchIntrv] 0.0424873 0.02993 1.42 0.1570 
JTAC _TLEmu 0.0418568 0.029924 1.40 0.1632 
C2_DecTime -0.038308 0.029928 -1.28 0.2018 
WCN_Rel -0.054672 0.028912 -1.89 0.0598 
(JTAC_SnsrRng-2750.01)*(JTAC_SnsrRng-2750.01) -0.250201 0.061826 -4.05 <.0001 
Tgt_C_Spd -0.4973 0.029918 -16.62 <.0001 

Figure 11. Scaled Estimates of the Regression Coefficients for the Stepwise Reduced 


Statistical Model with all Second Order Terms Considered. 


It should also be noted at this point that several of the terms listed do not appear 
to be statistically significant at the .05 level—indicated by a “Prob > Itl” value greater 
than 0.05. Specifically, the significance values of WCN Reliability, C2 Decision Time, 
and JTAC TLEmu fall well outside of this range. This effect is a result of statistical 
precedence, which indicates terms considered significant in higher order interactions 
should also be included in the model as main effects, regardless of their 


individual significance. 


Figure 12 shows the interaction profiles for three of the terms in the regression 
model. The vertical axis along the left side indicates the value of the response variable, 


in this case the proportion of target kills. The scale along the bottom indicates the range 


38 


of settings for the term listed in the respective column. The numbers on the interior of 
the plot correspond with the low and high settings for the term listed in that particular 
row. For example, the plot in the first row and second column displays the interaction 
between target speed and NEW sensor range. The values of 4 and 20 are the low and 
high settings for speed, respectively. The lines being nonparallel indicate an interaction 
between the terms is present in the response. Specifically, when the target’s speed is low, 
the response is minimally affected by increasing the NEW’s sensor range. If, however, 
the target speed is high, increases in the NEW’s sensor range have a significant impact on 
the proportion of target kills. This finding is consistent with the left branch of the 
partitioning tree shown previously in Figure 7. Also of interest is the large gap between 
the lines on the top row. This is indicative of the individual effect target speed has on the 


response. 
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Figure 12. Plot of Selected Interaction Terms in a Regression Model for Target Kills 
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As shown by the distribution in Figure 13, a large number of DPs resulted in zero 
target kills. This is of particular interest considering the data used for the analysis is 
summary data—meaning all 100 replications of the DP would have to result in missing 
the target for it to register as a proportion equal to zero. As may be expected from 
previous results, the distributions for JTAC sensor range and target speed shown in 
Figure 14 further illuminate their effect on target kills. The figure is generated from only 
the 119 DPs with a mean response of zero and illustrates a significant portion of the data 
contains low sensor ranges and high speeds. It appears obvious these two terms are 
combining to create a threshold where the simulation model indicates the system’s 
capability to identify, engage, and prosecute the target is unwieldy. A contour plot is 


often a useful tool to explore this possibility. 


Mean(C-Kills) 


119 of 257 total 
design points result 
in proportion of 
target kills = 0. 





Figure 13. Distribution of Target Kills for all 257 Design Points in the 
Single Target Scenario. 
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Tgt_C_Spd JTAC_SnsrRng 


Significant portion of the data which result in 
zero target kills occur for high target speed 
and low sensor range settings. 





Figure 14. Distributions of Target Speed and JTAC Sensor Range for 119 Design 
Points Where No Target Kills Occur. 


Figure 15 highlights the regions of the response where the combination of JTAC 
sensor range and target speed result in the gradations shown in the legend. The plot not 
only illustrates the combined effect of the two parameters, but also indicates a highly 


nonlinear aspect—as shown by the superimposed dotted line. 
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Contour Plot for Mean(C-Kills) 
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Figure 15. Contour Plot Highlighting the Proportion of Target Kills as a Function of 
the Combination Between JTAC Sensor Range and Target Speed. 


In order to better understand the combined effect of JTAC sensor range and target 
speed, and to more fully explore other potentially influencing parameters, a new variable 
was created by dividing sensor range by speed. The resulting variable has units of time 
(specifically seconds) and helps reveal previously undiscovered aspects of the results. In 
a physical sense, the variable represents the time it would take a target moving in a 
straight line directly at the JTAC to pass completely through its sensor range. To avoid 
introducing correlation effects into the statistical models, analysis using this new variable 
was accomplished with the original variables for JTAC sensor range and target 


speed excluded. 


The partitioning tree shown in Figure 16 includes the new variable and reveals 
insight into the effect of C2 decision time and launch aircraft speed. Within the context 
of factor ranges used in this scenario, the first two splits indicate threshold values for a 
notional time variable given by dividing the range and speed variables as discussed. The 
results indicate a time value exceeding 305 seconds produces favorable system 
performance—a proportion of target kills near 90%. If the time value falls between 
197 and 305 seconds, then the simulation model suggests C2 decision time and aircraft 


speed play a critical role in overall system performance. Specifically, when the decision 
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to initiate an attack sortie can be made in less than two minutes and the attack aircraft can 


fly to its launch standoff distance faster than 180 m/s (approximately 400 mph), then the 


estimate for proportion of kills increases from 55% to nearly 85%. 
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Figure 16. Partitioning Tree for Proportion of Target Kills with JTAC Sensor Range 
and Target Speed Combined Into One Variable. 
C; RESULTS FOR THE TWO-TARGET SCENARIO 


The second experiment involved the examination of NEW system effectiveness 
for the case of one mobile target, one dismounted high-value target (HVT), a 
ground-based weapon controller, and an aircraft equipped with two NEWs. The DoE 
included 27 factors, 257 design points (DP), and 100 replications of each DP. 


Results for system effectiveness against the mobile vehicle are very similar to the 
one target case, with strong influence of the outcome based on JTAC sensor range and 
target speed. For the HVT, though, the factors most influential on the proportion of kills 


are characteristics of the weapon itself. 
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Figure 17 illustrates the potential effect of NEW impact radius on a target moving 
at speeds characteristic of a dismounted patrol. The HVT speed for this experiment 
ranged from 0.5 to 4 m/s—the difference between sauntering around and running just 
under seven minute miles. Overall, the simulation results show an 88% proportion of 
HVT kills. For NEW impact radii exceeding 5.4 meters and probability of kill (Pk) 
greater than 0.92, the kill level approaches 95%. As a reminder, for this research, Pk 
indicates the likelihood of successful detonation upon impact. It can also be thought of as 
a conditional probability, meaning the probability a target is destroyed given its location 
is within the blast radius. On the left branch of the tree, the results show the precision of 
the NEW sensor (i.e., TLE) improves system performance if the blast radius is less than 
5.4 meters. The same explanation applies to the DPs for the larger impact radius, in cases 


where the Pk value is less than 0.92. 





























































































































































































































All Rows 
Count 257 
Mean 0.8852918 
Std Dev —_0.1983841 
NEW_ImpRad<5.4 NEW_ImpRad>=5.4 
Count 23 Count 234 
Mean 0.4891304 Mean 0.9242308 
Std Dev —_0.3898125 Std Dev —_0.1089889 
NEW_TLEmu>=2.832 NEW_TLEmu<2.832 NEW_PkA<0.918 NEW_PkA>=0.918 
Count 11 Count 12 Count 42 Count 192 
Mean 0.1318182 Mean 0.8166667 Mean 0.8328571 Mean 0.9442187 
Std Dev —_0.1686309 Std Dev 0.182624 Std Dev —0.1808555 Std Dev —0.0723315 
NEW_TLEmu<0.703 NEW_TLEmu>=0.703 
Count 5 Count 37 
Mean 0.56 Mean 0.8697297 
Std Dev —_0.3432929 StdDev —0.1114422 

















Figure 17. Partitioning Tree for Proportion of HVT Kills. 


The output generated for this research did not support an indication into whether a 
specific kill resulted from a redirected NEW or one launched specifically designated for 
the HVT. The next section, however, reveals insight regarding the effects of having only 


one NEW available for the attack sortie. 
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D. ADDITIONAL FINDINGS 


1. Results for Two Targets When Only One NEW is Available 


The results in this section were obtained using the same DoE as in the previous 
two-target case. The difference here is the fact that only one NEW was available on the 


attack aircraft. 


Figure 18 portrays the results for proportion of HVT kills. As previously noted, 
the intent of this analysis approach is to gain insight into which factors most significantly 
affect the response variable. Therefore, the reader is cautioned to not place undue 
influence on the value of the MOE itself, but instead to consider the relative changes in 


the response based on the specific value of the parameter under examination. 
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Figure 18. 


Partitioning Tree for Proportion of HVT Kills When 
Only One NEW is Available 


Unlike results for the one target case, decreases in JTAC sensor range tend to 


improve the value of the response. This is likely the influence of early detection in the 
simulation of the mobile vehicle target. If the JTAC’s sensors pick up the vehicle too 
soon, then the attack sortie is initiated and the resulting early launch of the NEW limits 


the ability of the weapon to respond to a request for redirect to the HVT. In fact, in some 
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cases, the NEW is actually flying in autonomous seeker mode by the time the priority 
target appears, and therefore nonresponsive to communications through the WCN. This 
logic is further supported by the results in Figure 18 for larger JTAC sensor ranges 
(i.e., left initial branching). Improvements in the subsequent split indicate a drastic 
difference in the MOE if the vehicle target is moving slow—thereby increasing the time 
before initial detection. As a final point on this figure, observe that even when 
considering the DPs where JTAC sensor range and vehicle target speed are working 
against HVT success, the priority target may still appear in time to initiate a NEW 


redirect if it materializes in the scenario less than 350 seconds after the start. 


46 


Vv. CONCLUSIONS 


A. ANALYSIS SUMMARY 


This research considered Network Enable Weapon (NEW) scenarios involving a 
ground-based Joint Terminal Attack Controller (JTAC), a single launch aircraft, and two 
target types. The targets were either characteristic of a mobile tracked vehicle or a 
dismounted patrol. The dismounted patrol (when present in the scenario) represented a 
high-value target (HVT) and, therefore, was considered a higher priority. The simulation 
model contained six agent types and allowed for the exploration of effects and 
interactions for up to 27 parameters, which, of course, would be very difficult to 


accomplish in a live test. 


For the scenario involving one mobile target, the results indicate a significant 
nonlinear interaction exists between the speed of the target and the sensor range of the 
JTAC. Specifically, for targets moving less than approximately 30 mph and a JTAC 
sensor range exceeding two kilometers, the model indicates an 80% improvement in the 
system’s ability to identify, track, engage, and kill the target. Within the ranges of 
parameter settings used in the design, these findings hold regardless of the other factor 
settings. JTAC sensor range and target speed were combined to create a parameter that 
notionally represents the time a target is observable by the ground controller. This 
approach revealed insight into the effect of Command and Control (C2) decision time and 
the aircraft’s speed. When the two factors combine to provide moderate target 
availability, system performance increases dramatically if the amount of time to respond 
to the Close Air Support (CAS) request is kept below two minutes and the strike aircraft 


proceeds to the weapon launch point at a speed greater than 400 mph. 


In scenarios involving the vehicle target and the HVT, the results vary depending 
on the weapons load of the strike aircraft. For an aircraft carrying two NEWs, the ability 
to successfully engage the HVT is highly dependent on the weapons’ impact radius and 
accuracy. When only one weapon is available, the ability to attack the HVT is 


significantly influenced by the timing of the scenario. For all designs used in this 
AT 


research, the vehicle target appears 10 seconds after the start of the simulation run. This 
means any attempts to prosecute a secondary target with only one weapon are likely to 
result from a redirected NEW. If the HVT appears too late in the scenario, or if the JTAC 
sensor range and vehicle target speed combine in a manner that results in an early CAS 
request, then the model indicates the system is unable to respond to a NEW redirect to 


the HVT. 


B. KEY INSIGHTS SUPPORTING USE OF AGENT-BASED MODELING 
(ABM) IN TEST PLANNING 


The overall objective of this research was to evaluate the use of ABM as a tool for 
enhancing joint test and evaluation. As a proof of concept, this thesis indicates ABM, 
combined with efficient experimental design, provides relatively quick insight into 
system-of-systems performance for a NEW construct. The simulation model can easily 
be tailored for specific scenarios. When paired with a variety of designs, a myriad of 
factor settings can quickly be explored in order to gain insight into specific areas 


of interest. 


Conducting actual live tests for a NEW scenario will involve months, if not years, 
of planning, and cost several millions of dollars.49 Using ABM techniques before 
conducting such events to understand potential interactions and system capabilities is 
well worth the investment. When properly combined with the right simulation tools, the 
concepts of efficient experimental design allow for an investigation of system 
performance across a broad spectrum of parameters and factor settings. Considering the 
broad uncertainties and complex interactions typical of military conflict, ABM offers a 
worthwhile analytical approach to explore elements of warfare that might otherwise be 


too costly or time intensive to study by other means. 


49 Based on estimates provided by JTEM for an August 2007 planned test event. 
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C. RECOMMENDATIONS FOR FUTURE RESEARCH 


This research is the first known attempt to investigate the application of ABM and 
efficient experimental design in support of JTEM test planning. The overall concept 
appears promising, but there are many aspects of the tools and techniques used in this 


thesis yet to be fully explored. 
1. Experimental Design 


The designs in this thesis were structured to quickly explore the key interactions 
in a NEW system-of-systems construct. Many of the initial experiments were used to 
error-check the model—making it critical to insure the simulations could be conducted in 
a short amount of computer processing time on one machine. Additionally, once the 
model was deemed sufficiently stable for conducting analysis, there was limited time to 
fully explore a broader range of parameters and parameter settings. Future efforts should 
consider increasing the number of replications for each design point beyond the 100 used 
in this thesis. Also, there are advances to consider in the application of nearly 
orthogonal, space-filling techniques, offering an increase in the number of factors and 
factor levels. Each of these approaches will likely result in longer computer run times for 


the experiment, but may reveal additional insights as well. 


The results indicate a nominal increase in the predictive power of the regression 
formula when higher-order effects are added to the main effects. This suggests a 
sequential screening design may be useful in developing an initial estimate of which 
factors to consider in the design. The space-filling attributes of the Nearly Orthogonal 
Latin Hypercube (NOLH) design, however, are still required to gain insight into the 


complex interactions among the factors. 
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2. 


Simulation Model 


The simulation model is essentially a prototype constructed to facilitate a proof of 


concept investigation of ABM applied to test planning. There are many enhancements 


capable of improving the breadth of applications to the NEW construct, a few of which 


are provided here. 


Use of an Unmanned Aerial Vehicle (UAV) to identify and track 
targets and act as the third-party weapon controller: The current 
research was limited to a ground-based controller. The extension to an 
aerial based controller opens the possibility of increased observation time 
on the target may be one approach to addressing the issue of high-speed 
targets. 


Launch aircraft providing In-Flight Target Updates (IFTUs): 
Currently, the model only supports in-flight targeting updates from the 
JTAC. In addition to adding IFTU capability to the launch aircraft, 
another extension would involve conditions in which the NEW is 
designated for direct attack mode—meaning the weapon goes directly into 
an autonomous mode and engages the target with its own sensors upon 
launch. 


NEW redirect logic: A simplified approach was used to determine 
whether or not the NEW could comply with a request to redirect to a 
different target. A more thorough physics-based representation is 
recommended to more accurately portray actual weapon performance 
capabilities. 


Terrain effects: This research considered operations on a flat surface. 
Adding terrain and/or building features to the model will increase analysis 
into the issues of line-of-sight limitations. 


Target awareness: In a real-world context, targets may become aware 
they are being tracked and potentially targeted and will likely alter their 
behaviors upon such recognition. Advancements to the model should 
support an analysis into this behavior. 


JTAC FOV: For both ground-based and UAV controllers, the sensors are 
typically constrained to a specific field of view (FOV). This research 
considered a 360-degree FOV, which is a somewhat unrealistic limitation. 


Randomized factor settings: “Scripted” settings for events such as target 
initial locations and waypoints were used for this research. Randomizing 
these and other factor settings may reveal additional insights and more 
accurately represent real-world dynamics. Other factors to consider for 
random effects include target pop-up time, number of targets, target speed, 
and target duration. 
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Fratricide and collateral damage: Operations in urban or constrained 
environments are prevalent in today’s warfare. Modeling the potential 
impact to unintended targets, whether friendly or neutral, may provide a 
useful measure of system effectiveness. 


Target identification to kill output measure: Including time from target 
identification to kill may be a useful additional measure. 


Scenarios and Factors 


There are obviously numerous scenarios possible for both real-world NEW 


engagements and joint test events. While a limited set were explored in this research, the 


simulation model is easily adaptable to a wide range of possibilities. 


Time Sensitive Targeting: Target disappears once route is complete. 


Launch aircraft orbiting: Rather than have the launch aircraft always 
initiate from a set point in space, using an orbit might reveal effects 
dictated by the size, type, or direction of the orbit pattern. 


Kill criteria: Different target sets may not require catastrophic 
destruction. In such cases, the concept of obtaining a mobility or 
firepower kill may be a sufficient objective and should be modeled 
accordingly. 


Multiple targets: As in real life, an engagement may contain multiple 
targets and targets of various types and priorities. An additional extension 
would be to add “legacy” weapons that aren’t network enabled to the 
aircraft load and investigate the effects of decisions regarding which 
weapon type to use against a specific type of target or target behavior. 


JTAC Pd toggle: As indicated in Chapter III, the JTAC’s set detection 
probability toggles to 1.0 (meaning certainty) once a target is initially 
detected. This was done to simulate a focused attention on a target, but 
may not always be realistic. Conducting model runs in which the Pd 
remains at the setting for a particular design point (DP) throughout the 
engagement may reveal different results. 


Process 


There are several aspects to the model design and implementation process that 


could be improved or expanded. Possibly the most beneficial expansion would be to 


include even more discussions with subject matter experts (SMEs) regarding the model’s 


functionality and capabilities. The “face validity” gained by exposing the model to SME 
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input and evaluation would increase the credibility of the results and move the entire 


process further from an academic endeavor and into an operationally beneficial product. 


From a usability perspective, the process of initiating model runs and generating 
model output could be improved to enhance the portability of the product. As it currently 
stands, the model requires extensive user understanding of computer programming 
interfaces in order to design and implement a specific experiment. Additionally, the 
output generates a massive amount of data files that must be processed with tools 


secondary to the model itself—generating another layer of analytical complexity. 


It is anticipated that students and faculty from NPS will continue working with 
JTEM to expand on this and other research efforts supporting joint test planning. The 
author was requested to comment on the general usefulness of the current model in 


addressing JTEM requirements. The responses are provided in Appendix D. 
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APPENDIX A. SUMMARY DESCRIPTION OF PARAMETERS IN 
THE SIMULATION MODEL 


This table provides a summary description of the factors in the simulation model. 
It can be used as a means of reference when viewing figures in the analysis to better 
understand the link between factor name, what that factor represents, and the units 


of measure. 


Description 
TAC Sensor Range 
TAC Prob Detection for HVT 

TAC Prob Detect for Mobile Vehicle Target 


TAC_MTDF TAC Multi-Target Degradation Factor 


TAC_IFTU_Intrvl |JTAC In-flight Target Update Interval 


NEW_Spd EW Speed 


INEW_ImpRad___ [NEW Impact Radius 
INEW_SnstRng [NEW Sensor Range 
NEW Mean Target Location Error 
NEW Target Location Error Standard Deviation 
NEW Prob Detection for HVT n/a 
NEW Prob Detection for Mobile Vehicle Target 
NEW Prob Kill for HVT 

NEW Prob Kill for Mobile Vehicle Target 

HVT Speed 
HVT Pop-up Time 
Target C Speed 
Target C Pop-up Time 
IAC Spd Aircraft Speed 


Aircraft Number of Launch Attempts 

Aircraft Interval Between Launch Attempts 
Weapon Control Network Latency n/a 
Command and Control Decision Time 
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APPENDIX B. EXPERIMENTAL DESIGN FOR THE SINGLE 
TARGET SCENARIO 
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Design Pt # | Tgt_C_Spd | NEW_Spd | NEW_ImpRad | NEW_SnsrRng | NEW_TLEmu | NEW_TLEsigma | NEW_PdC | NEW_PkC | AC_Spd | AC_LnchDist | AC_UpdtReq 
i 13.5 178 
2 17.44 183 
3 5.88 198 
4 10.38 184 
5 13.81 196 
6 19.31 192 
Z 11.06 171 
8 9.88 196 
D, 18.88 163 
10 17.81 143 
11 11.75 166 
12 5.19 162 
13 14.06 164 
14 15.5 150 
15 4.63 147 
16 8.31 164 
17 16.75 194 
18 16.81 172 
19 11.56 189 
20 8.44 185 
21 16.56 188 
22; 13.75 194 
23) 11.94 171 
24 9.31 185 
25 16.69 153 
26 18.5 163 
Zl 11.69 170 
28 8.13 160 
29 18.38 152 
30 12.5 152 
31 7 168 
32 8.06 157 
33 17.38 178 
34 16 191 
35 9.25 192 
36 11.38 187 
37 12.56 199 
38 14.94 192 
SY) 10.31 187 
40 8.88 189 
41 19.75 168 
42 17.88 159 
43 75 164 
44 6.94 158 
45 14.25 150 
46 17.63 155 
47 6.06 160 
48 7.56 168 
49 14.31 200 
50 19.81 198 
Sl 7.81 196 
av 5 193 
23) 15.81 185 
54 13.06 184 
59. 4.38 186 
56 7.69 182 
Sill 16.06 156 
58 15.75 166 
59 7.75 141 
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64 5.94 144 
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67 9.81 192 
68 5.38 179 
69 16.13 173 
70 17.13 179 
71 6.75 200 
12; 5.69 191 
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22.8 
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00.5 
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29 
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Low 4 140 3 1000. 0 0 0.85 0.9 154 18500 10 
High 20 200 30 3000 3 2 1 1 254 74000 45 
Decimals 2 0 1 0 2 2 2; 2 1 0 0 
Design Pt # | Tgt_C_Spd | NEW_Spd | NEW_ImpRad | NEW_SnsrRng | NEW_TLEmu | NEW_TLEsigma | NEW_PdC | NEW_PkC | AC_Spd | AC_LnchDist | AC_UpdtReg | 
82 18.44 175 5.1 2352 0.74 0.87 0.97 0.96 184.1 19801 26 
83 10.56 174 6.8 2578 0.64 0.9 0.87 0.9 232.5 55355 33 
84 11.25 198 12.5 2836 1.11 0.64 0.94 0.98 170.4 51670 38 
85 15.38 190 15 2930 0.08 0.59 0.92 0.92 241.5 44299 24 
86 13.88 175 12.4 2648 0.66 1.4 0.97 1 154.4 49502 18 
87 4.88 181 3.2 2453 1.09 1.01 0.9 0.93 238 27172 29 
88 10 183 ee 2313 0.92 0.03 0.96 0.97 175.1 41480 38 
89 17.31 157 9/5 2992 2.32 0.7 0.94 0.91 218.1 21752 35 
90 13.31 163 129 2773 1.58 1.54 0.9 1 198.5 44732 43 
91 8.94 143 7.6 2617 1.97 13 0.94 0.94 220.8 57957 22 
92 8.56 163 5.8 2086 1.52 0.34 0.87 0.98 193.1 60125 19 
93 15.31 170 11 2547 0.82 0.8 0.99 0.94 225.9 58607 28 
94 13.25 146 8.3 2148 125. 1.33 0.93 0.97 165.7 62727 28 
95 10.88 165 12.8 2820 1.64 1.9 0.93 0.91 207.9 45166 16 
96 4.5 167 5.3 2867 2:3) 0.76 0.91 0.99 188.4 31725 24 
oT 16.38 180 26.6 1586 0.51 0.89 0.91 0.96 209.1 33676 13 
98 13.19 181 25.8 1844 2.34 1.64 0.95 0.93 167.7 28473 25 
99 5.44 187 27 1859 0.94 1.88 0.87 1 236 61209 32 
100 5.06 195 28.6 1063 1.27 0.23 0.97 0.95 181.7 71182 36 
101 19.94 198 21.6 1336 1.04 0.27 0.87 0.96 244.6 60992 19 
102 18.19 189 28.7 1836 0.47 1.99 1 0.92 177.8 62293 23 
103 7.06 184 18.8 1617 2.4 1.09 0.89 0.97 232.1 40180 36 
104 11.31 179 21.4 1570 1.29 0.44 0.95 0.93 203.6 28039 43 
105 19.44 140 29.5, 1578 1.07 0.45 0.99 0.97 209.9 19150 29 
106 17.19 147 17.9 1766 1.95 1.94 0.85 0.93 171.6 39963 43 
107 8.38 151 19.1 1109 2.48 1:37. 0.94 0.96 220.4 52104 11 
108 95 167 19.8 1750 1.66 0.51 0.89 0.91 205.2 55789 15 
109 14.56 154 20.9 1984 1.02 0.77 0.93 0.99 231.3 52754 41 
16.8 1195 1 1.38 0.88 0.95 172.8 63160 40 
18.2 1820 2.36 1.92 0.98 0.95 225.1 23053 24 
26.1 1906 1.39 0.54 0.86 0.94 202.4 22836 18 
12.2 1953 1.86 0.88 0.88 0.99 226.7 41914 11 
11.8 1156 1.41 1.23 0.99 0.95 192.3 36061 14 
8.7 1281 0.2 1.57 0.87 0.96 229.4 49936 37 
5.2 1703 1.35 0.68 0.99 0.91 158.3 68580 44 
3.9 1891 0.35 0.73 0.86 0.96 236.8 71832 17 
hl 1695 0.68 1.85 0.96 0.94 160.6 41697 24 
10.4 1016 2.42 1.17 0.9 0.98 247 46684 40 
12.3 1305 0.37 0.79 0.94 0.9 201.7 30857 34 
3.1 1484 1.93 0 0.94 0.97 241.9 27389 28 
6.1 1359 0.21 1.12 0.87 0.92 184.5 20234 34 
8.2 1344 2.21 1.18 0.95 0.99 214.5 68797 21 
15.3 1211 1.84 0.78 0.92 0.91 197.4 67063 14 
9.1 1594 2.03 0.24 0.93 0.98 231.7 57740 38 
14.5 1391 0.8 1.78 0.9 0.93 191.5 48852 28 
5.4 1727 0.59 175 0.98 0.99 212.6 22402 24 
4.8 1172 1.99 0.13 0.92 0.93 162.6 39313 20 
16.5 2000 2.5 1 0.93 0.95 204 46250 28 
Eye 1000 0.63 1.63 0.96 0.99 235.6 55139 31 
15.7. 1055 0.78 0.91 0.91 0.9 165.3 45600 44 
8.1 1117 0.55 0.71 0.97 0.99 239.5 21969 16 
9.9 1242 0.25 L7 0.86 0.9 158.7 38229 25 
134 10.19 144 8.6 1492 1.56 1.87 0.99 0.97 246.2 46033 39 
135 4.69 148 53): 1531 0.23 0.55 0.87 0.92 183.3 23270 30 
136 12.94 169 16 1469 2.07 0.66 1 0.97 222 65979 13 
137 14.13 144 10 1602 1.6 1.03 0.93 0.92 188.8 50369. 13 
138 13; 177 8.8 1523 0.1 1.65 0.86 0.95 211 63811 22 
139 6.19 197 13.5 1867 2.25 0.09 0.94 0.92 159.1 66629. 25 
140 12.25 174 16.4 1047 2.01 0.2 0.9 0.98 213.8 38012 41 
141 18.81 178 12 1461 1.89 1.02 0.93 0.9 190.3 34109 30 
142 9.94 176 8.5 1711 1.68 1.26 0.89 1 206.7 28906 22 
143 8.5 190 8.4 1023 2.44 0.57 0.95 0.93 188 40613 15 
144 19.38 193 15.8 1477 2.19 0.52 0.92 1 239.2 67930 34 
145 15.69 176 10.5 1125 0.72 1.79 1 0.92 196.2 64244 32 
146 7.25 146 20 1719 7; 1.69 0.92 0.99 221.2 55572 33 
147 TA9 168 22.4 1297 1.62 0.45 0.86 0.94 191.1 47551 39 
148 12.44 151 28.9 1258 1.05 0.65 0.96 0.98 244.2 35627 27 
149 15.56 155 22.7 1445 0.41 1.07 0.88 0.91 173.1 31074 15 
150 TA4 152 28.1 1031 0.18 1.25 1 0.99 237.2 22619 35 
isl 10.25 146 26.3 1266 0.88 0.47 0.89 0.91 155.6 47117 35 
152 12.06 169 16.9 1438 1:37, 0.63 0.97 0.98 218.5 57523 11 
153 14.69 155 28 1367 1.72 1.05 0.93 0.93 179.8 59908 21 
154 7.31 187 19.6 1930 0.02 1.86 0.91 0.97 248.5 51236 18 
155 5.5 177 28.8 1633 0.57 0.48 0.97 0.95 157.9 57307 27 
156 12.31 170 18.1 1789 0.96 0.56 0.9 0.97 208.3 43215 30 
157 15.88 180 20.1 1609 2.29 1.05 0.92 0.94 173.5 19584 37 
158 5.63 188 19 1328 1.13 1.02 0.89 0.98 216.1 26955 17 
159 115 188 2550 1500 2.13 0.2 0.98 0.92 180.2 38445 31 
160 17 172: 26.8 1656 0.45 0.58 0.9 0.96 248.1 67279 39 
161 15.94 183 19.2 1922 0.33 1.58 0.94 0.94 164.2 61859 43 
162 6.63 162 3 2797 1.46 1.34 0.98 0.94 211.4 66195 39 
163 8 149 3.7 2063 2.11 0.19 0.91 0.98 179 52971 37 
164 14.75 148 46 2625 2.15 0.84 0.97 0.92 245 43432 15 
165 12.63 153 6.3 2492 0.16 1.14 0.87 0.98 195 43865 19 
166 11.44 141 9.6 2586 0.84 1.16 0.99 0.91 208.7 47984 43 
167 9.06 148 10.2 2813 2.09 0.85 0.85 0.98 169.6 42781 40 
168 13.69 153 9.3 2039 1.54 0.49 0.98 0.92 221.6 63377 15 
169 15.13 151 11.1 2484 1.43 1.06 0.89 0.95 180.6 67713 11 
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Low 1 10 500. 10 2 0.7 15 0.95 0 10 
High 5 30 5000 100 20 1 45 1 Di 200 
Decimals 0 1 0 0 0 2 1 2 2; 1 
Design Pt # | AC_LnchAtmpts | AC_LnchIntrvl | JTAC_SnsrRng | JTAC __TLEmu | JTAC_TLEsigma | JTAC_PdC | JTAC_IFTU_Intrvl | WCN_Rel | WCN_Lat | C2_DecTime 
82 4 11.9 1959 42 18 0.72 35.6 0.98 0.7 126.5 
83 2 20.9 2363 31 17 0.82 38.8 0.98 0.03 157 
84 2 20.2 1889 17 19 0.77 38.6 0.99 0.82 167.3 
85 4 15.2 4719 95 7 0.83 32 0.99 0.6 154 
86 4 14.1 3225 81 5 0.75 35.4 0.98 0.27 128.8 
87 2 26.1 2697 70 4 0.71 36.9 0.98 0.95 131 
88 3 26.7 4367 78 8 0.73 33) 1 0.62 143.6 
89 5 20.2 2838 19 14 0.9 20 0.98 0.53 142.9 
90 4 11.3 3295 21 13 0.9 19:5 0.98 0.51 116.1 
Gil 2 29.2 3945 51 16 0.92 2241 1 0.22 152.5 
Gye 2 26.9 2855 34 15 0.99 17.5 0.99 0.21 145.8 
93 4 12.6 2029. 62 10 0.97 21.7 1 0.18 161.4 
94 3 10.4 1063 90 8 0.95 21.3 1 0.23 181.4 
95 1 22.2 1713 92 9 0.86 15.9 0.99 0.04 111.7 
96 2 24.9 1994 99 9 0.9 29:3 1 0.13: 175.5 
97 2 27.7 1695 71 15 0.87 36 1 1.24 82 
98 1 21.3 1854 68 13 0.88 26 0.99 1.8 75.3 
99 5 16.3 1256 58 16 0.94 36.7 0.99 1.76 76.1 
100 4 16.5 1221 70 18 0.85 39.6 0.99 1.03 19.6 
101 2: 22.3 4191 13 5 0.89 34.8 1 1.15 62 
102 3 25.2 3875 44 8 0.97 29.4 0.99 1.05 76.8 
103 5 18.1 3365 46 9 0.85 31.1 0.99 1.84 73.1 
104 4 14.4 4051 34 5 0.93 42.8 0.99 1.22 73.8 
105 3 24.5 4701 63 17 0.82 19.3 1 1.7 24.1 
106 3) 28.4 4561 56 12 0.79 26.7 1 0.83 64.9 
107 4 18.9 2785 93 16 0.81 19.6 0.98 1.69 30 
108 5 14.6 4736 50 16 0.76 19.8 0.98 1.86 29: 
109 1 22 2012 49 7 0.7 18.4 0.98 1.33 47.1 
111 3 13:5 887 35 9 79 DED 0.99 1:37. 41.2 
113 5 21.6 1484 84 19 0.98 36.6 0.97 0.74 39.7 
115 3) 10.3 518 69 17 42.7 0.95 0.66 17.4 
117 3 25.8 3857 41 7 0.96 37.7 0.97 0.65 78.3 
119 2 17 3436 18 6 93 42.3 0.97 0.57 96.8 
121 4 29.5 3717 98 14 23.8 0.97 0.41 46.4 
123) 2 13.6 2434 88 16 0.79 22.6 0.97 0.25 49.3 
124 0.33 36.7 
126 5 21.4 2504 28 10 72 16.6 0.97 0.44 53.8 
128 2 13.2 1326 37 ‘L 87 29.6 0.97 0.11 10.7 
130 3 28.8 4227 78 15 94 43.5 0.99 1.52 194.8 
132 3 11.4 3910 86 19 32.1 0.99 1.12 164.4 
134 4 27.7 816 24 5 0.9 0.99 0.84 188.9 
135 3 21.2 992 18 2 0.99 33 0.99 1.3 174 
136 1 16.8 1414 14 5 0.88 32.2 0.99 1.71 151.8 
137 2: 13.4 1941 14 3 0.92 41.5 0.98 1.85 176.3 
138 3 24.1 2223 94 16 0.73 27.3: 0.99 1.27 185.2 
139 3 27.8 1291 1D: 12 0.77 30.5 0.99 1.42 189.6 
140 2: 19.6 1080 99 11 0.78 17.9 1 1.8 119.8 
141 3 14 2240 72 15 0.78 25.3 0.99 1.99 115.4 
142 4 28.4 3031 24 8 0.74 20.9 0.99 1.93 138.4 
143 3: 23.1 3014 13 9 0.72 16.4 1 1.31 178.5 
144 2 10.9 4455 51 10 0.73 21.8 0.99 1.63 194.1 
145 2 15.4 2873 41 4 0.7 15-7 0.98 1.19 116.9 
146 2. 27 3576 89 13 0.83 4L.7 0.96 0.06 103.5 
147 il 26.3 2820 2B 14 0.91 44.4 0.97 0.77 86.4 
148 3 14.7 4947 91 17 0.94 30.8 0.95 0.88 LTE 
149 4 15.8 4156 91 20 0.96 37.3 0.97 0.66 113.9 
150 1 28.2 1467 38 3 0.89 et) 0.97 0.55 88.7 
151 3 25.2 1730 11 7 0.89 37.6 0.96 0.34 168.1 
152 5 12.4 2943 27 fs 0.92 45 0.97 0.9 114.6 
1153) 5 10.5 1379 39 8 0.86 43.8 0.96 0 147.3 
154 2 23.6 1396 61 18 0.71 24.5 0.96 0.87 159.9 
155 3 22.1 3049 58 19 0.77 25 0.96 0.8 112.4 
156 4 14.9 1818 74 12 0.74 19.2 0.96 0.16 139.9 
Sai 4 18.8 1432 87 17 0.72 155 0.95 0.89 193.3 
158 3) 19.5 3822 36 8 0.77 20.6 0.96 0.17 191.1 
159 1 23.4 2908 43 7 0.76 17.1 0.96 0.05 99.8 
160 4 15 3400 39 3 0.75 25.9 0.95 0.38 120.6 
161 3, 11.7 4385 49 3 0.81 23 0.95 0.23 149.5 
162 4 16 4314 36 16 0.74 32:5, 0.98 1:73; 25.6 
163 > 10.9 4789 56 12 0.85 36.3 0.96 1.21 23.4 
164 1 22.9 3893 25 13 0.75 43.1 0.96 1.6 90.9 
165 1 24.5 4807 Bf: 14 0.74 30.9 0.97 1.41 32.3) 
166 4 15.6 1748 88 4 0.79 34.6 0.98 1.91 108.7 
167 4 1527: 1572 87 7 0.86 41.1 0.98 1.74 70.9 
168 2 29.4 676 64 11 0.83 35.2 0.95 1.58 27.1 
169 2 23.8 1977 96 6 0.76 34.3 0.97 1.65 26.3 
















































































































































































































































































Low 4 140 3 1000 0 0 0.85 0.9 154 18500 10 
High 20 200 30 3000 5 2 1 1 254 74000 45 
Decimals 2, 0 i 0 2; 2 DD 2) il 0 0 
Design Pt # | Tgt_C_Spd | NEW_Spd | NEW_ImpRad | NEW_SnsrRng | NEW_TLEmu | NEW_TLEsigma | NEW_PdC | NEW_PkC | AC_Spd | AC_LnchDist | AC_UpdtReg | 
170 4.25 172 10.1 2570 1.45 1.67 0.9 0.94 218.8 68146 23 
171 6.13 181 147 2219 191 0.05 0.93 0.98 185.3 64678 20 
172 16.5 176 3.6 2008 0.27 0.18 0.9 0.93 212.2 29557 34 
173 17.06 182 9.2 2336 1.88 1.48 0.94 0.95 156.3 29990 26 
174 9.75 190 12.6 2594 2.46 1.61 0.89 0.92 250.5 34326 29 
175 6.38 185 3.3 2602 0.49 0.4 0.99 0.98 206 24137 24 
176 17.94 180 94 2055 2.05 0.28 0.93 0.93 207.1 71398 38 
177 16.44 172 47 2445 2.27 1.59 0.96 0.95 190.7 71615 42 
178 9.69 140 20.3 2258 2.38 1.98 0.99 0.93 243.5 59258 42 
179 4.19 142 26 2438 0.39 0.8 0.85 0.97 204.8 73566 4B 
180 16.19 144 26.5 2922 07 0.23 0.97 0.93 214.2 39096 13 
181 19 147 24 2461 1.76 1.68 0.85 1 164.9 38879 10 
182 8.19 155 29.6 2859 2.23 1.19 0.96 0.91 223.1 19367 39 
183 10.94 156 26.4 2727 1.74 0.53 0.89 I 200.5 36277 41 
184 19.63 154 24.1 2031 0.04 041 0.97 0.91 233.3 67496 23 
185 16.31 158 25 2852 0.76 1.93 0.89 0.96 154 54922 26 
186 7.94 184 17.4 2227 1.21 15 0.91 0.91 238.8 66412 20 
187 8.25 174 215 2914 0.9 0.55 0.95 0.96 182.5 70314 10 
188 16.25 199 19.3 2117 0.86 0.59 0.88 0.94 245.8 18717 30 
189 15.19 186 21.8 2266 0.53 1.62 0.99 0.96 174.3 18500 34 
190 4 179 25.6 2188 1.48 1.95 0.85 0.92 241.1 32809 14 
191 4.44 199 23.3 2680 0.98 0.02 0.95 0.96 194.6 25654 22 
192 19.06 186 21.1 2766 1.82 0.16 0.86 0.94 246.6 59475 45 
193 18.06 196 17.6 2203 0.43 1.84 0.97 0.99 192.7 51453 33 
194 8.75 169 10.9 1250 5 17 0.98 0.98 154.8 72049 29 
195 7.13 145 14.4 1313 4.86 0.73 0.91 0.91 252.8 69014 37 
196 14.19 148 14.6 1219 471 0.29 0.94 0.99 160.3 38662 in 
197 18.63 161 3.8 1102 4.39 1.28 0.87 0.91 239.9 36494 21 
75 1625 3.77 1.63 0.95 0.99 181 26738 37 
143 1094 3.67 0.48 0.92 0.93 228.6 35844 45 
113 1828 3.83 O17 0.98 0.96 1774 70965 20 
10.7 1641 3.5 1.66 0.86 0.94 224.3 60342 27 
10.8 1039 3.69 131 0.88 0.95 197.8 59041 13 
13.3 1898 2.83 0.96 0.96 0.9 229.8 62076 14 
45 1805 4.88 0.27 0.86 0.96 173.9 27605 36 
13.1 1758 3.85 1.96 0.97 0.94 252 377195 32 
16.3 1672 3.22 1.89 0.88 0.95 156.7 31941 12 
5.6 1977 4.94 0.92 0.96 0.93 222.4 20018 23 
14.1 1875 3.81 0.61 0.87 0.99 157.1 57090 38 
15.2 1289 4.69 174 0.95 0.95 231 44082 36 
17.1 1680 3.2 1.91 0.93 0.97 167.3 50152 43 
27.9 1648 4.26 1.13 0.88 0.94 223.9 72699 29 
26.2 1422 4.36 1.1 0.98 1 175.5 37145 22 
20.5 1164 3.89 1.36 0.91 0.92 237.6 40830 17 
214 3 150 18 1070 4.92 141 0.93 0.98 166.5 48201 31 
13 20.6 1352 4.34 0.6 0.88 0.9 253.6 42998 37 
29.8 1547 3.91 0.99 0.95 0.97 170 65328 26 
25.9 1688 4.08 1.97 0.89 0.93 232.9 51020 17 
23.5 1008 2.68 13 0.91 0.99 189.9 70748 20 
25.1 1227 3.42 0.46 0.95 0.9 209.5 47768 12 
25.4 1383 3.03 07 0.91 0.96 187.2 34543 33 
27.2 1914 3.48 1.66 0.98 0.92 214.9 32375 36 
22 1453 418 12 0.86 0.96 182.1 33893 27 
293 10.75 194 24.7 1852 3.15 0.67 0.92 0.93 242.3 29773 27 
224 13.13 175 20.2 1180 3.36 0.1 0.92 0.99 200.1 47334 39 
225 19.5 173 27.1 1133 2.7 1.24 0.94 0.91 219.6 60775 31 
226 7.63 160 64 2414 4.49 inn 0.94 0.94 198.9 58824 42 
227 10.81 159 72 2156 2.66 0.36 0.9 0.97 240.3 64027 30 
228 18.56 153 6 2141 4.06 0.12 0.98 0.9 172 31291 23 
229 18.94 145 4.4 2938 3.73 1.77 0.88 0.95 226.3 21318 19 
230 4.06 142 114 2664 3.96 1.73 0.98 0.94 163.4 31508 36 
231 5.81 151 43 2164 453 0.01 0.85 0.98 230.2 30207 32 
232 16.94 156 14.2 2383 2.6 0.91 0.96 0.93 175.9 52320 19 
233 12.69 161 11.6 2430 3.71 1.56 0.9 0.97 204.4 64461 12 
234 4.56 200 3.5 2422 3.93 1.55 0.86 0.93 198.1 73350 26 
235 6.81 193 15.1 2234 3.05 0.06 I 0.97 236.4 52537 12 
236 15.63 189 13.9 2891 2.52 0.63 0.91 0.94 187.6 40396 44 
237 14.5 173 13.2 2250 3.34 1.49 0.96 0.99 202.8 36711 40 
238 9.44 186 12.1 2016 3.98 1.23 0.92 0.91 176.7 39746 14 
239 9.19 174 16.2 2805 4 0.62 0.97 0.95 235.3 29340 15 
240 16.63 195 14.8 2180 2.64 0.08 0.87 0.95 182.9 69447 31 
241 18.69 196 69 2094 3.61 1.46 0.99 0.96 205.6 69664 37 
242 9.38 148 20.8 2047 3.14 1.13 0.97 0.91 181.3 50586 44 
243 11.81 149 21.2 2844 3.59 0.77 0.86 0.95 215.7 56439 41 
244 B 147 24.3 2719 48 0.43 0.98 0.94 178.6 42564 18 
245 17.69 143 27.8 2297 3.65 1.32 0.86 0.99 249.7 23920 im 
246 431 159 29.1 2109 4.65 1.27 0.99 0.94 171.2 20668 38 
247 11.19 143 25.3 2305 4.32 0.15 0.89 0.96 2474 50803 31 
248 13.56 165 22.6 2984 2.58 0.83 0.95 0.92 161 45816 15 
249 13.94 159 20.7 2695 4.63 1.21 0.91 I 206.3 61643 21 
250 9 199 29.9 2516 3.07 2 0.91 0.93 166.1 65111 27 
251 4.75 173 26.9 2641 4.79 0.88 0.98 0.98 223.5 72266 21 
252 13.38 176 24.8 2656 2.79 0.82 0.9 0.91 193.5 23703 34 
253 14.88 177 177 2789 3.16 1.22 0.93 0.99 210.6 25438 41 
254 6 180 23.9 2406 2.97 1.76 0.92 0.92 176.3 34760 17 
255 65 171 18.5 2609 42 0.22 0.95 0.97 216.5 43648 27 
256 18.25 174 27.6 2273 441 0.25 0.87 0.91 195.4 70098 31 
257 19.19 191 28.2 2828 3.01 1.88 0.93 0.97 245.4 53188 35 
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Low 1 10 500. 10 2 0.7 15 0.95 0 10 
High 5 30 5000 100 20 1 45 1 2) 200 
Decimals 0 1 0 0 0 2 1 zi 2; 1 
Design Pt # | AC_LnchAtmpts | AC_LnchIntrvl | JTAC_SnsrRng | JTAC __TLEmu | JTAC_TLEsigma | JTAC_PdC | JTAC_IFTU_Intrvl | WCN_Rel | WCN_Lat | C2_DecTime 

170 4 15.9 2768 32 11 0.87 23.9 0.95 1.66 80.5 
171 B) 19.1 2521 50 12 0.86 28.8 0.97 1.98 43.4 
172 2 20.7 S317. 33 19 0.98 29.9 0.96 1.32 40.4 
173 2 23 922 40 12 0.95 20.2 0.95 1.29 18.2 
174 5 16.1 3629 67 10 0.92 18.2 0.96 1.51 35.2: 
175 4 17.4 3330 59 3 0.94 28.1 0.96 1.72 47.9 
176 1 26.2 3084 66 10 0.88 30.2 0.97 1.91 61.2 
177 3 24.7 2574 94 8 0.95 19.7 0.97 0.92 84.2 
178 1 18.2 3664 26 18 0.78 39 0.99 0.72 27.8 
179 2 10.2 4332 45 17 0.72 31.6 0.98 0.75 63.4 
180 4 25.9 4402 21 15 0.76 35.7 0.99 0.02 92.4 
181 4 PARE 3313 10 11 0.83 36.4 0.98 1.02 79.8 
182 2: 12.5 1801 80 8 0.82 40.9 0.99 1.04 56.8 
183 2 19.7 623 93 4 0.84 38.4 0.98 0.08 54.5 
184 5 28.5 746 80 5 0.72 42 0.98 0.84 91.6 
185 5 273 588 57 12 0.8 33.9 0.99 0.54 22.6 
186 i 17.5 605 47 18 0.93 16.1 0.98 0.63 85.7 
187 3 19.4 1766 15 13 0.86 18.8 0.97 0.73 74.6 
188 3 28.8 535 25 17 0.97 22.5 0.99 0.91 38.9 
189 3 22.3 2152 48 17 0.99 28.6 0.98 0.45 TS. 
190 3 17.3 4543 60 2. 0.89 25.8 0.99 0.1 13 

191 2 11.1 5000 33, 4 1 17.8 1 0.86 10 

192 4 26.6 3102 98 6 0.96 17.6 0.98 1.02 51.6 
193 4 20.8 3189 66 8 26.6 0.99 0.59 44.9 
194 5 29.9 4490 67 8 19 1 0.52 87.2 
195 4 29.8 4666 65 6 24.7 0.99 0.13 13.7 
196 2 14.5 3242 90 6 17 0.98 0.46 37.5 
197 3 12 2293 719 6 28.7 0.98 0.52 107.2 
198 4 24.8 2346 15 18 25.1 0.97 0.55 87.9 
199 3. 21 1150 23 19 15.1 0.99 0.45 99.1 
200 3 12.1 852 25 20 32.3 1 0.64 14.5 
201 2 14.5 2047 28 19 18.6 1 0.8 109.5 
202 3 21.6 658 83 2 38.1 0.98 0.47 38.2 
203 5 22.7 2117 63 i 40.1 0.98 0.43 104.3 
204 2 16.2 975 56 2 39.5 0.99 0.56 50.8 
205 2 11 1027 83 9 37.1 0.99 0.41 30.8 
206 3 26.3 4631 34 15 43.7 0.99 0.48 69.4 
207 4 215: 4297 48 20 44.2 0.99 0.19 12.2 
208 2 17.2 4262 46 11 44.8 0.99 0.36 59.7 
209 3 16.6 3172 16 16 39.3 0.98 0.61 102 

210 2 28 3840 TD. 4 26.3 0.95 17. 67.9 
211 2 28.1 3541 68 4 24.4 0.97 1.3 83.5 
212 1.97 53 

213 4 19.8 3611 93 3 21.4 0.96 1.18 42.7 
214 2 24.8 781 15 15 28 0.96 1.4 56 

215 2; 25.9 2275 29 17 24.6 0.97 1.73 81.3 
216 4 13.9 2803 40 18 23.1 0.97 1.05 79 

217 3) 13.3 1133 33 14 26.5 0.95 1.38 66.4 
218 1 19.8 2662 91 8 40 0.97 1.47 67.1 
2A) 2, 28.7 2205 89 9 40.5 0.97 1.49 93.9 
220 4 10.8 1555 59 6 37.9 0.95 1.78 57.5 
221) 4 13.1 2645 76 7 42.5 0.96 1.79 64.2 
222) 2 27.4 3471 48 12 38.3 0.95 1.82 48.6 
223 3 29.6 4438 20 14 38.7 0.95 L777 28.6 
224 5 17.8 3787 18 13 44.1 0.96 1.96 98.3 
225) 4 15.1 3506 11 13 30.7 0.95 1.88 34.5 
226 4 12.3 3805 33 7 24 0.95 0.76 128 

Q2iT, 5 18.7 3646 42 9 34 0.96 0.2 134.7 
228 1 23.7 4244 52 6 23.3 0.96 0.24 133.9 
229 2 23.5 4279 40 4 20.4 0.96 0.97 190.4 
230 4 17.7 1309 97 17 25.2 0.95 0.85 148 

231 3 14.8 1625 66 14 30.6 0.96 0.95 133.2 
232 1 21.9 2135 64 13 28.9 0.96 0.16 136.9 
233 2 25.6 1449 76 17 17.2 0.96 0.78 136.2 
234 3 15.5 799 47 > 40.7 0.95 0.3 185.9 
235 3 11.6 939 54 10 33.3 0.95 1.17 145.1 
236 2. 21.1 2715 17 6 40.4 0.97 0.31 180 
237 1 25.4 764 60 6 ; 40.2 0.97 0.14 188.1 
238 B) 18 3488 61 15 1 41.6 0.97 0.67 162.9 
239, 3 19.5 4139 98 11 0.86 35.9 0.96 0.05 165.9 
240 3 26.5 4613 55 13 0.91 32.8 0.96 0.63 168.8 
241 2 20.1 4596 81 18 0.91 39.7 0.97 0.77 144.3 
242 1 18.4 4016 26 3 0.72 23.4 0.98 1.26 170.3 
243 2 13 2961 10 9 0.83 16.8 0.97 1.09 137.7 
244 4 29.7 4982 41 5 0.79 17.3) 1 1.34 192.6 
245 4 23.3 2891 42 2 0.8 25.5 0.97 1.68 141.4 
246 3 14.2 1643 69 16 0.74 22.3 0.98 1.35: 131.7 
247 2 12.9 641 82 19 0.74 27.4 1 1.07 180.7 
248 4 23 2064 92 16 0.77 170 0.98 1.43 113.2 
249 B) 22.4 729 53 13 0.76 26.4 0.98 1.61 151 

250 2 10.5 1783 12 8 0.95 36.2 0.98 1:59 163.6 
251 2: 18 2381 35 4 0.99 31.8 0.98 0.99 127.3 
252 4 26.4 3066 22 6 0.91 37.4 0.98 175: 160.7 
253) b) 27.2 1502 35 11 0.86 44.6 0.98 1.67 173.3 
254 1 10.7 3980 88 12 0.91 38.9 1 1.88 177 

255 i 18.6 2996 82 12 0.98 43.4 0.98 1.56 156.2 
256 a 25.7 3594 80 19 1 36.8 0.98 1.06 198.5 
257 4 26.8 4174 13: 15 0.83 30.4 0.98 1.89 199.3 


























APPENDIX C. CALCULATING AN EXPECTED VALUE OF THE 
RESPONSE VARIABLE BASED ON REGRESSION COEFFICIENTS 


The two tables below show the regression coefficients and corresponding scaled 
estimates for a regression model with five terms (plus the intercept term). If given the 
parameter coefficients (top table), the formula for calculating an estimate of the response 
is simply obtained by adding the intercept term to the sum-product of the regressor terms 
and their corresponding assigned value. For example, by selecting values for the five 
terms in this model (within the ranges of the original design) and inserting them into the 
formula below, an estimate of the response is obtained—in this case, the proportion of 


mobile vehicle target kills. 


(-0.424164) + (-0.062232 * Tgt_C_Spd) + (0.0048507 * NEW_ImpRad) + 
(0.9302622 * NEW_PdC) + (0.001617828 * AC_Spd) + 0.0001395 * JTAC_SnsrRng) 


If given the scaled estimates, a conversion must be made to obtain the original 
parameter estimates. As indicated in the bottom figure, the estimates are centered by 
mean and scaled by range/2. This indicates the scaled estimate was obtained by 
multiplying the original coefficient by the value of (range of factor setting)/2. For 
example, in the design, target speed varied between 4 and 20 m/s. This provides a 
range/2 value of (20—4)/2 = 8. Multiplying the original coefficient estimate for target 


speed by 8 gives the value of the scaled estimate (rounding considered): 


(- 0.062232) * 8 = —0.497856. 





Parameter Estimates 








Term Estimate Std Error tRatio  Prob>itl 
Intercept -0.424164 0.42449 -1.00 0.3186 
Tgt_C_Spd -0.062232 0.004021 -15.48 <.0001 
NEW_ImpRad 0.0048507 0.002382 2.04 0.0428 
NEW_PdC 0.9302622 0.427881 2.17 0.0306 
AC_Spd 0.0016178 0.000643 2.51 0.0125 
JTAC_SnsrRng 0.0001395 0.000014 9.75 <.0001 





Scaled Estimates 


Continuous factors centered by mean, scaled by range/2 





Term Scaled Estimate Std Error t Ratio Prob>itl 
Intercept 0.4830739 0.018645 25.91 <.0001 
Tgt_C_Spd -0.497853 0.032169 -15.48 <.0001 
NEW_ImpRad 0.0654849 0.032162 2.04 0.0428 
NEW_PdC 0.0697697 0.032091 2.17 0.0306 
AC_Spd 0.0808914 0.032169 2.51 0.0125 
JTAC_SnsrRng 0.3137775 0.032169 9.75 <.0001 
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APPENDIX D. ANNOTATIONS REGARDING APPLICABILITY OF 
MODEL IN ADDRESSING OVERALL JTEM REQUIREMENTS 


J TEM Requirement Resolution within "TheTester" 


Description Developer Comments 


| escription 
Ability to model J TEM Test 07 and J TEM Test 08 
scenario and its associated entities. The ABM 
should be designed to model at least the J TEM Test 
07 and J TEM Test 08 scenarios 


P - | don't know what these are yet to make a further 
assessment. We have Aircraft (Launch Platform), NEW, 
Legacy, C2ISRNetwork (assigns CAS missions to aircraft 
based on Target classification and priority and weapon- 
target pairing), and WCNetwork (models latency and 
Task Force organizations to support the scenario reliability of target updates getting from J TAC to NEW) 
The Army’s None-Line-Of- Site Launch System 
(NLOS-LS) platform with its Precision Air Munitions |N - although if the NLOS-LS can be modeled using the 
(PAMs), including ability to vary platforms, PAM current Aircraft Agent, than Y. We'd need more 
mix, and PAM range information on what constitutes this package of systems. 


The Air Force’s Networked Enabled Weapon (NEW) 
and its delivery air platforms, including ability to Y - this was the canonical scenario. Again, depends on 
vary platforms, NEW mix, and NEW range what is included in the "delivery air platforms" 


Y. Stationary and mobile Targets are modeled, with each 

having a target class that can represent different types of 

targets. For mobile targets, you specify a path anda 
Appropriate enemy targets (including types/speeds |speed. The target class is also linked to Pd and Pk for the 
of targets) to support the test scenario J TAC/NEW sensors and the NEW 


N. Would need to know the different ways jamming effects 
es scene eee systems ,e.g., does it change the Pd, add noise to the 
Ability of enemy to jam PAM and NEW systems TLE, other mechanisms? 

P. User can add any number of agents (other than 
seins tote untatecoroane eaten by adding them to the scenario file. 
Ability to model unit order of battle Would need more testing, though. 


P. Only material, personnel, and organizations, and 
personnel and organizations only by number. Would need 
to add different types of doctrinal "behaviors" to capture 
Able to vary Blue and Red materiel, doctrine, this fully. | think this is one area where we'd need to 
organizations, personnel, and leadership discuss future efforts 
Clutter (e.g., civilian personnel, air traffic) to 
evaluate its impact on system performance 
operations (e.g., deconfliction of air space for N. Would need more information on what needs to be 
launching the NLOS-LS or NEW) modeled and what interactions there are 
N. Certainly something to be done for future effort. Would 
need some design work so that conceptual models could 
Ability to add conceptual models of new systems easily be "plugged-in" 
N/P. ROEs, in a limited sense, are "hard-wired". The J TAC 
senses a Target, "knows" it is one, and then sends CAS 
request, which then gets prosecuted for that target. 
There is a Redirect by the J TAC if a target of higher 
priority presents itself. Currently, the only thing | would 
Ability to model rules of engagement (ROEs) call an ROE that is implemented. 


Pe 
Joint Command and Control (JC2) entities with at 
least J oint Tactical Air Control (J TAC) elements 
(ability to vary types) and a representation of the 
J oint Force Commander and staff. For the 
purposes of this ABM effort, this latter 
representation should be kept simple, providing a 
minimal number of representatives (e.g., the J oint 
Force Commander, the Joint Force Air Component |N/P - not sure how modeling the Commanders will impact 
Commander, the J oint Force Land Component the system. Will need more information on this to answer 
Commander, and Joint Force Effects Coordination fully. However, the current C2I1SRNetworkAgent only 
Cell) to accomplish JC2 with respect to the NLOS- |models latency in decision time, and the decision modeled 
LS and NEW systems is the assignment of aircraft to a CAS mission. 
ES 





J TEM Requirement 


Resolution within "TheTester" 


Identifier Description Developer Comments 


2 


3 


Ability to model/assess J C2 


Model/assess shared situational awareness, 
planning, conduct of netcentric operations, and C2 
efficiency 


Ability to conduct battle damage assessment 
Consider NATO Code of Best Practices for C2 
Assessment, published by the Command and 
Control Research Program (CCRP) under the 
auspices of the Assistant Secretary of Defense 
(C31), 2002 


Ability to model/assess sensor systems 


N. Would need some kind of conceptual model to start 
with. The other question, is what part of all of THAT 
system affects the system under test? In the larger sense 
of modeling systems of systems, | guess it depends on 
what's getting tested and where the connections are, and 
what of the larger system affects those connections. 

N. BDA is not performed, nor information from a BDA 
inserted into the overall system. 


N/P. The only set of "sensors" in the model are those 
attached to the JTAC and the NEW 


3.1 Unattended ground sensors 


3.2 


Sensors mounted on unmanned ground and air 
platforms 


3.3 Sensors on NLOS-LS/PAM and NEW systems 


3.4 


Sensors for NLOS operator and J TAC (including 
target location system) 


Ability to model/assess the J oint Fires (including 
Joint Close Air Support) and JC2 Doctrine/TTPs to 
support the above scenarios, include ability to 


4.1 Vary doctrine/TTPs 
4.2 Manage the air space 


4.3 


N 
i N 


Model impacts on Circular Error Probable (CEP) for 
weapon systems 


P - Have included TLE mu/sigma for the NEW 


4.3.1 _|NEW update rates Y 


NEW-Link16 latency 

JTAC effectiveness (beam blocked, beam split, 
environmental effects) 

Relative CEP 


C2 issues 


Ability to model/assess netcentric communications 
systems, including the ability to: 


Y - Latency and Reliability of the link are modeled using 
uniform distributions 


P - Only to the extent that the J TAC is tracking the target 
using the sensor, that includes TLE mu/sigma factors 

N - Model uses bivariate Normal TLE approximation 

P - Addresses latency associated with C2 time to respond 
to CAS request 


5.1 Track message traffic N 


Degrade system effectiveness with effects of 
weapons and environment 


N 


6 Ability to model the environment, including 


Mai Representation of the terrain where tests will occur |N - How does terrain affect the system? 


5.2 
1 


6.3 


Features (e.g., buildings, foliage) and weather, or 
at least the effects that features and weather have 
on system of systems performance and TTPs 
Visibility 


N/P - Currently modeled by varying the Pd and TLE of the 
JTAC or NEW 
N/P - Same as 6.2 


6.4 Civilians 
7 Ability to mode/assess some human dimensions 


For message traffic received by a human, add 
ability to adjust the understanding of the message 
(this may be as simple as “understand” and “does 
not understand”) 

Variable operating skills for the JTAC and NLOS 
personnel 

Civilian relationships to Blue and Red 

Ability to affect human performance (e.g., 
information/message overload slows down decision 
processes or causes an action to be missed 





altogether) 
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N 


N/P - Throught the Pd/TLE and IFTU frequency currently 
N - Neutrals currently not modeled 





N/P - Currently, only in C2 decision latency 
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